Run more of the test suite in parallel closes #610 #612

Merged

simon merged 11 commits from issue/mitgliederverwaltung-610 into main

2026-10-19 21:13:44 +02:00

simon commented

2026-10-19 21:03:16 +02:00

Owner

closes #610

Description of the implemented changes

The changes were:

Bugfixing
New Feature
Breaking Change
Refactoring

The suite spent about 235 s of its ~280 s in the serial phase. This branch flips the slowest serial modules (selected by measurement) to async: true, after giving every async test its own values under unique indexes so concurrent sandbox transactions no longer wait on each other's uncommitted rows. Tests that touch process-global state stay serial, and each serial module now states why in a # Serial: <reason>. comment that an architecture test enforces. No product code changed.

What has been changed?

Give each async test its own values under unique indexes
Run the measured slow test modules async
Keep log-capturing tests in serial modules
Give each async OIDC test its own identity subs
Share the unauthenticated and linking conn helpers via ConnCase
Run the OIDC sign-out tests that seed the discovery cache serially
State why each serial test module runs serially
Run the cycle generator edge case tests async
Require a serial reason and no log capture in async modules
Add the rules that keep async tests deterministic
Document the async-test safety checklist and new suite timings

Definition of Done

Code Quality

No new technical depths
Linting passed
Documentation is added were needed

Accessibility

New elements are properly defined with html-tags
Colour contrast follows WCAG criteria
Aria labels are added when needed
Everything is accessible by keyboard
Tab-Order is comprehensible
All interactive elements have a visible focus

Testing

Tests for new code are written
All tests pass
axe-core dev tools show no critical or major issues

Additional Notes

Before / after (`just test-fast --seed <s>`, same 4-vCPU host)

Seed	Before total / async / sync	After total / async / sync	Host context before → after
101	285.1 / 45.0 / 240.0 s	231.2 / 54.9 / 176.2 s	another worktree's tests running → idle
202	278.6 / 44.1 / 234.4 s	228.6 / 55.6 / 172.9 s	another worktree's tests running → idle
303	281.5 / 46.0 / 235.5 s	277.8 / 54.7 / 223.0 s	another worktree's tests running → same

The async phase grows by ~10 s and the sync phase shrinks by ~60 s on an idle host (about −50 s total, ~18 %). Under concurrent load from another test run the total gain mostly disappears (~1 %), because the larger async phase competes for the same CPUs. Final just ci-dev: 231.4 s (53.0 s async, 178.4 s sync), 0 failures.

Async-test rules introduced

Two distinct 40P01 mechanisms: FK MultiXact locks (fixed earlier by deferrable FKs) and unique-index waits on another sandbox transaction's uncommitted row (fixed here by System.unique_integer([:positive]) values per test).
Settings-row updates stay inside the sandbox in test (no SettingsCache is started), but they hold the row lock until the test ends; flipping such a module from serial to async needs a measured gain and an isolation proof (done for cycle_generator_edge_cases_test.exs).
ScopeOverviewParityPropertyTest runs async with @moduletag timeout: 180_000: lock sampling showed no lock waits, its duration grows with CPU load (up to 48 s under load).

Handover to #540 (serial modules held back by global state)

OS env (System.put_env/delete_env): config, SMTP, OIDC, Vereinfacht and release tests — needs config passed explicitly instead of System.get_env/1.
Application env (Application.put_env/delete_env): join notifier, OIDC provider observer, mailer, join confirm — needs injected module/config.
Settings row updates (Membership.update_settings/2) with no measurable gain (<1 s), plus the LiveView modules deferred to U7/U10/#549 — needs settings passed explicitly or a per-test settings fixture.
Named SettingsCache process, Logger level, Mix shell, telemetry handlers, the OIDC discovery cache (:persistent_term) and the JoinRateLimit ETS table — need process-local alternatives or stay serial.
DDL / migration tests and shared files (PDF template, temp dirs) — stay serial or get per-test paths.

Review

The review loop reached its 4-iteration cap (11 → 2 → 2 → 4 findings, all accepted, no security findings); the remaining findings were addressed and the loop was closed by the reviewer's approval.

Process

Backtrack from tdd to requirements: the scope was extended from test/mv/** to test/mv_web/** outside live/ after the first measurement showed test/mv/** held only 38.9 s of 235 s serial time.

Larger commits

"Give each async test its own values under unique indexes" (41 files, ~1.1k lines added) and "State why each serial test module runs serially" (156 files, comments only) are large but mechanical.

closes #610 ## Description of the implemented changes The changes were: - [ ] Bugfixing - [ ] New Feature - [ ] Breaking Change - [ ] Refactoring The suite spent about 235 s of its ~280 s in the serial phase. This branch flips the slowest serial modules (selected by measurement) to `async: true`, after giving every async test its own values under unique indexes so concurrent sandbox transactions no longer wait on each other's uncommitted rows. Tests that touch process-global state stay serial, and each serial module now states why in a `# Serial: <reason>.` comment that an architecture test enforces. No product code changed. ## What has been changed? - Give each async test its own values under unique indexes - Run the measured slow test modules async - Keep log-capturing tests in serial modules - Give each async OIDC test its own identity subs - Share the unauthenticated and linking conn helpers via ConnCase - Run the OIDC sign-out tests that seed the discovery cache serially - State why each serial test module runs serially - Run the cycle generator edge case tests async - Require a serial reason and no log capture in async modules - Add the rules that keep async tests deterministic - Document the async-test safety checklist and new suite timings ## Definition of Done ### Code Quality - [ ] No new technical depths - [x] Linting passed - [x] Documentation is added were needed ### Accessibility - [ ] New elements are properly defined with html-tags - [ ] Colour contrast follows WCAG criteria - [ ] Aria labels are added when needed - [ ] Everything is accessible by keyboard - [ ] Tab-Order is comprehensible - [ ] All interactive elements have a visible focus ### Testing - [x] Tests for new code are written - [x] All tests pass - [ ] axe-core dev tools show no critical or major issues ## Additional Notes ### Before / after (`just test-fast --seed <s>`, same 4-vCPU host) | Seed | Before total / async / sync | After total / async / sync | Host context before → after | |---|---|---|---| | 101 | 285.1 / 45.0 / 240.0 s | 231.2 / 54.9 / 176.2 s | another worktree's tests running → idle | | 202 | 278.6 / 44.1 / 234.4 s | 228.6 / 55.6 / 172.9 s | another worktree's tests running → idle | | 303 | 281.5 / 46.0 / 235.5 s | 277.8 / 54.7 / 223.0 s | another worktree's tests running → same | The async phase grows by ~10 s and the sync phase shrinks by ~60 s on an idle host (about −50 s total, ~18 %). Under concurrent load from another test run the total gain mostly disappears (~1 %), because the larger async phase competes for the same CPUs. Final `just ci-dev`: 231.4 s (53.0 s async, 178.4 s sync), 0 failures. ### Async-test rules introduced - Two distinct `40P01` mechanisms: FK MultiXact locks (fixed earlier by deferrable FKs) and unique-index waits on another sandbox transaction's uncommitted row (fixed here by `System.unique_integer([:positive])` values per test). - Settings-row updates stay inside the sandbox in test (no `SettingsCache` is started), but they hold the row lock until the test ends; flipping such a module from serial to async needs a measured gain and an isolation proof (done for `cycle_generator_edge_cases_test.exs`). - `ScopeOverviewParityPropertyTest` runs async with `@moduletag timeout: 180_000`: lock sampling showed no lock waits, its duration grows with CPU load (up to 48 s under load). ### Handover to #540 (serial modules held back by global state) - OS env (`System.put_env/delete_env`): config, SMTP, OIDC, Vereinfacht and release tests — needs config passed explicitly instead of `System.get_env/1`. - Application env (`Application.put_env/delete_env`): join notifier, OIDC provider observer, mailer, join confirm — needs injected module/config. - Settings row updates (`Membership.update_settings/2`) with no measurable gain (<1 s), plus the LiveView modules deferred to U7/U10/#549 — needs settings passed explicitly or a per-test settings fixture. - Named `SettingsCache` process, Logger level, Mix shell, telemetry handlers, the OIDC discovery cache (`:persistent_term`) and the `JoinRateLimit` ETS table — need process-local alternatives or stay serial. - DDL / migration tests and shared files (PDF template, temp dirs) — stay serial or get per-test paths. ### Review The review loop reached its 4-iteration cap (11 → 2 → 2 → 4 findings, all accepted, no security findings); the remaining findings were addressed and the loop was closed by the reviewer's approval. ### Process Backtrack from tdd to requirements: the scope was extended from `test/mv/**` to `test/mv_web/**` outside `live/` after the first measurement showed `test/mv/**` held only 38.9 s of 235 s serial time. ### Larger commits "Give each async test its own values under unique indexes" (41 files, ~1.1k lines added) and "State why each serial test module runs serially" (156 files, comments only) are large but mechanical.

simon added 11 commits

2026-10-19 21:03:16 +02:00

test: give each async test its own values under unique indexes ddd2555f13

test: run the measured slow test modules async 0ee42477c9

test: keep log-capturing tests in serial modules 7e3360a2f1

test(auth): give each async OIDC test its own identity subs f9fa420f09

test(auth): share the unauthenticated and linking conn helpers via ConnCase e6c6ce3496

test(auth): run the OIDC sign-out tests that seed the discovery cache serially 340096d59c

test: state why each serial test module runs serially 8f054d9680

test(membership-fees): run the cycle generator edge case tests async a5256577eb

test(architecture): require a serial reason and no log capture in async modules 330d27ccbb

docs(guidelines): add the rules that keep async tests deterministic 3872ec576d