Run the LiveView test files in parallel #611

New issue

Closed

opened 2026-10-19 13:47:38 +02:00 by simon · 0 comments

simon commented

2026-10-19 13:47:38 +02:00

Owner

Motivation

#610 measured where the serial time of the fast suite is spent (main after #605, a662fe40, per-module medians of three --slowest-modules runs, 4-vCPU dev VM; numbers in the #610 PR):

Area	Serial files	Serial time
`test/mv_web/live/member_live/`	47	131.6 s
other `test/mv_web/live/`	25	39.1 s
`test/mv/**` (#610 scope)	78	38.9 s
`test/mv_web/` outside `live/`	14	36.0 s (#610 scope, see below)

A plain fast-suite run takes about 280 s (45 s async, 235 s sync). The LiveView files hold about 170 s of the 235 s sync time. #610 deliberately does not flip them, because #602 (theme/buttons), U7 (search), U10 (list toolbars) and #549 (saved views) change the same files. Flipping them one by one inside those issues would deliver the gain only at the end of the UI roadmap; every pipeline gate until then pays the slow suite. This issue flips them in one go, right after #602 is merged and before U4–U7, so the remaining UI work already runs on the faster suite.

Estimate (not measured): moving ~170 s of serial time into the async phase saves about 100–120 s per fast-suite run on 4 vCPUs (moved work still costs about a third in the async phase).

Prerequisites

#610 merged: the 40P01 deadlock class is fixed (unique values under unique identities in every async file, including live/**), the rules are in CODE_GUIDELINES.md §4, and the guard test enforces a reason on every async: false and no capture_log in async files.
#602 merged (it edits tests under live/** and test/support/).

Approach

Follow the #610 procedure (measurement, per-file isolation proof, batch proofs, evidence log in .pipeline/followups/flaky-tests.md):

Quick win first: member_live/index_test.exs (26.5 s, 61 tests). It is serial only because of one @slow performance test with a wall-clock budget ("boolean filter performance with 150 members"). ci-dev excludes :slow anyway. Move that test into a small serial module (reason: "wall-clock budget, CPU contention makes it flaky"), flip the rest.
Measure the serial live/** modules (--slowest-modules, three seeds) and pick candidates by the #610 rule (Pareto ≥ 80 % of the serial live/** time, median ≥ 1 s).
Flip per directory batch (member_live/, then the other live/ directories), one file per commit, each with --repeat-until-failure 5 and five seeded full runs per batch. Before flipping, fix what breaks under concurrency in LiveView tests specifically: whole-page substring assertions (render(view) =~ …, refute html =~ …) → has_element?, assert_flash/3, assert_field_error/3; shared fixed values; reads compared in order without a sort.
Notable files:
- user_live/form_test.exs (8.8 s) still carries the #533 note "deferred scope decision" — flip.
- member_live/index_value_control_test.exs (8.6 s), index_sort_atom_property_test.exs (8.3 s), index_field_visibility_test.exs (6.7 s, #533 note) — largest after index_test.
- The nine member_live/index_groups_*, index_custom_fields_display, show_groups_display files with the #533 note.
- global_settings_live_test.exs (5.1 s) mutates the Settings singleton → stays serial here; #540 makes it isolatable.
Every file that stays serial keeps a one-line reason (the #610 guard enforces it).

Coordination

Run after #602, before U4–U7. Conflicts with U4–U7 are limited to the use … async: line and test data; resolve on merge.
#609 (test hardening) sweeps whole-page substring assertions suite-wide. If #609 runs first, step 3 gets cheaper; if this issue runs first, it fixes the class in the files it flips and #609 skips them.
#540: global_settings_live_test.exs and the settings-dependent tests.

Acceptance criteria

member_live/index_test.exs runs async except the moved performance test.
Every flipped file passed --repeat-until-failure 5 and its batch passed five seeded full runs, one while other pipeline work ran in parallel; runs listed in the evidence log.
No whole-page substring assertions, no capture_log, no fixed values under a unique identity in flipped files.
Before/after fast-suite numbers (total, async, sync; three runs each; same host; seeds and parallel context recorded) in the PR description.
just ci-dev and just typecheck green.

Out of scope

Flipping files that need global state (Settings singleton, application or OS env) — #540.
Speeding up individual slow tests.

## Motivation #610 measured where the serial time of the fast suite is spent (main after #605, `a662fe40`, per-module medians of three `--slowest-modules` runs, 4-vCPU dev VM; numbers in the #610 PR): | Area | Serial files | Serial time | |---|---|---| | `test/mv_web/live/member_live/` | 47 | 131.6 s | | other `test/mv_web/live/` | 25 | 39.1 s | | `test/mv/**` (#610 scope) | 78 | 38.9 s | | `test/mv_web/` outside `live/` | 14 | 36.0 s (#610 scope, see below) | A plain fast-suite run takes about 280 s (45 s async, 235 s sync). The LiveView files hold about 170 s of the 235 s sync time. #610 deliberately does not flip them, because #602 (theme/buttons), U7 (search), U10 (list toolbars) and #549 (saved views) change the same files. Flipping them one by one inside those issues would deliver the gain only at the end of the UI roadmap; every pipeline gate until then pays the slow suite. This issue flips them in one go, right after #602 is merged and before U4–U7, so the remaining UI work already runs on the faster suite. Estimate (not measured): moving ~170 s of serial time into the async phase saves about 100–120 s per fast-suite run on 4 vCPUs (moved work still costs about a third in the async phase). ## Prerequisites - #610 merged: the 40P01 deadlock class is fixed (unique values under unique identities in every async file, including `live/**`), the rules are in `CODE_GUIDELINES.md` §4, and the guard test enforces a reason on every `async: false` and no `capture_log` in async files. - #602 merged (it edits tests under `live/**` and `test/support/`). ## Approach Follow the #610 procedure (measurement, per-file isolation proof, batch proofs, evidence log in `.pipeline/followups/flaky-tests.md`): 1. **Quick win first: `member_live/index_test.exs` (26.5 s, 61 tests).** It is serial only because of one `@slow` performance test with a wall-clock budget ("boolean filter performance with 150 members"). `ci-dev` excludes `:slow` anyway. Move that test into a small serial module (reason: "wall-clock budget, CPU contention makes it flaky"), flip the rest. 2. **Measure** the serial `live/**` modules (`--slowest-modules`, three seeds) and pick candidates by the #610 rule (Pareto ≥ 80 % of the serial `live/**` time, median ≥ 1 s). 3. **Flip per directory batch** (`member_live/`, then the other `live/` directories), one file per commit, each with `--repeat-until-failure 5` and five seeded full runs per batch. Before flipping, fix what breaks under concurrency in LiveView tests specifically: whole-page substring assertions (`render(view) =~ …`, `refute html =~ …`) → `has_element?`, `assert_flash/3`, `assert_field_error/3`; shared fixed values; reads compared in order without a sort. 4. Notable files: - `user_live/form_test.exs` (8.8 s) still carries the #533 note "deferred scope decision" — flip. - `member_live/index_value_control_test.exs` (8.6 s), `index_sort_atom_property_test.exs` (8.3 s), `index_field_visibility_test.exs` (6.7 s, #533 note) — largest after `index_test`. - The nine `member_live/index_groups_*`, `index_custom_fields_display`, `show_groups_display` files with the #533 note. - `global_settings_live_test.exs` (5.1 s) mutates the Settings singleton → stays serial here; #540 makes it isolatable. 5. Every file that stays serial keeps a one-line reason (the #610 guard enforces it). ## Coordination - Run after #602, before U4–U7. Conflicts with U4–U7 are limited to the `use … async:` line and test data; resolve on merge. - #609 (test hardening) sweeps whole-page substring assertions suite-wide. If #609 runs first, step 3 gets cheaper; if this issue runs first, it fixes the class in the files it flips and #609 skips them. - #540: `global_settings_live_test.exs` and the settings-dependent tests. ## Acceptance criteria - `member_live/index_test.exs` runs async except the moved performance test. - Every flipped file passed `--repeat-until-failure 5` and its batch passed five seeded full runs, one while other pipeline work ran in parallel; runs listed in the evidence log. - No whole-page substring assertions, no `capture_log`, no fixed values under a unique identity in flipped files. - Before/after fast-suite numbers (total, async, sync; three runs each; same host; seeds and parallel context recorded) in the PR description. - `just ci-dev` and `just typecheck` green. ## Out of scope - Flipping files that need global state (Settings singleton, application or OS env) — #540. - Speeding up individual slow tests.

simon added this to the Optimize CI and Dev Environment | TI I milestone

2026-10-19 13:47:38 +02:00

simon added this to the Sprint 18: Oktober 2026 project

2026-10-19 13:47:38 +02:00

simon referenced this issue from a pull request that will close it,

2026-10-20 23:23:17 +02:00

Run the LiveView test files in parallel closes #611 #616

simon closed this issue

2026-10-20 23:37:17 +02:00

simon referenced this issue from a commit

2026-10-20 23:37:17 +02:00

Merge pull request 'Run the LiveView test files in parallel closes #611' (#616) from issue/mitgliederverwaltung-611 into main