Test-suite: fix existing flakiness + cut runtime #533

New issue

Closed

opened 2026-06-15 22:12:53 +02:00 by moritz · 0 comments

moritz commented

2026-06-15 22:12:53 +02:00

Owner

Motivation

The suite is both slow (~206s; 164.6s of it synchronous) and already flaky (known: unrelated async
failures appear on re-run). Root cause is currently unknown, so this issue starts with diagnosis and then both
fixes the flakiness and reduces runtime.

Scope

Diagnose existing flakiness (first step). Run the suite repeatedly with varying --seed; capture which
tests fail and on which seeds; find the root cause. Usual Elixir suspects: async tests sharing global state
(Application env, named ETS/Agent, Gettext locale), Process.sleep-based waits instead of event-based
assertions, PubSub/Task timing, DB-sandbox ownership, order-dependence.
Fix the identified flaky tests so they pass deterministically (isolate their state, or make waits
event-based with assert_receive/polling — no sleeps).
R-023 — cut runtime. Flip genuinely-isolated test files to async: true (the bulk of the 164.6s sync);
remove the Process.sleep(1000) import-wait (import_live_test.exs:40, itself a flakiness vector); drop the
dead seeds guard (seeds_test.exs:36). Anchors also: index_field_visibility_test.exs:13,161.
R-024 — decouple. Introduce shared create_fee_type/create_cycle fixtures (currently duplicated 18×/8×);
replace :sys.get_state(socket) internal-state assertions (index_test.exs:15-27,1014) with behavior
assertions.

Acceptance criteria

The previously-flaky tests pass deterministically across N seeded runs (e.g. 10×).
No Process.sleep-based waits remain in the touched tests.
Suite runtime materially reduced (report before/after).
just ci-dev green.

Caution

Flipping a genuinely shared-state test to async: true can introduce flakiness — convert only files with
isolated data, per-file, and re-run repeatedly to confirm. Re-run before treating any newly-async failure as a
real regression.

## Motivation The suite is both **slow** (~206s; 164.6s of it synchronous) and **already flaky** (known: unrelated async failures appear on re-run). Root cause is currently unknown, so this issue starts with diagnosis and then both fixes the flakiness and reduces runtime. ## Scope - **Diagnose existing flakiness (first step).** Run the suite repeatedly with varying `--seed`; capture which tests fail and on which seeds; find the root cause. Usual Elixir suspects: async tests sharing global state (Application env, named ETS/Agent, Gettext locale), `Process.sleep`-based waits instead of event-based assertions, PubSub/Task timing, DB-sandbox ownership, order-dependence. - **Fix the identified flaky tests** so they pass deterministically (isolate their state, or make waits event-based with `assert_receive`/polling — no sleeps). - **R-023 — cut runtime.** Flip genuinely-isolated test files to `async: true` (the bulk of the 164.6s sync); remove the `Process.sleep(1000)` import-wait (`import_live_test.exs:40`, itself a flakiness vector); drop the dead seeds guard (`seeds_test.exs:36`). Anchors also: `index_field_visibility_test.exs:13,161`. - **R-024 — decouple.** Introduce shared `create_fee_type`/`create_cycle` fixtures (currently duplicated 18×/8×); replace `:sys.get_state(socket)` internal-state assertions (`index_test.exs:15-27,1014`) with behavior assertions. ## Acceptance criteria - The previously-flaky tests pass deterministically across N seeded runs (e.g. 10×). - No `Process.sleep`-based waits remain in the touched tests. - Suite runtime materially reduced (report before/after). - `just ci-dev` green. ## Caution Flipping a genuinely shared-state test to `async: true` can *introduce* flakiness — convert only files with isolated data, per-file, and re-run repeatedly to confirm. Re-run before treating any newly-async failure as a real regression.