Tests: remove the known causes of random red CI runs #695

Closed
opened 2026-11-19 13:42:04 +01:00 by simon · 0 comments
Owner

Description

Random failures block merges: the Renovate pull request #689 failed on CI in a test without a relation to the update, and every dependency update (e.g. plug 1.20.4) runs the same risk. Every developer and every pipeline run re-runs suites "until green", which hides real failures.

The failures are not random. The evidence log (flaky-tests.md) shows five classes, each with a cause in the test setup:

  1. Sandbox ownership shorter than the test timeout. test/mv_web/member_export/scope_overview_parity_property_test.exs allows 180 s (@moduletag timeout: 180_000); the sandbox keeps DBConnection's default ownership_timeout of 120 s. On a slow CI host the property loses its connection (DBConnection.Holder.checkout … no process). Seen on #689, Drone build 3096, seed 919006.
  2. Global stderr capture in async tests. test/mv_web/components/core_components_button_api_test.exs (lines ~80–86) asserts an empty capture_io(:stderr) while test/mv_web/components/feedback_callout_test.exs:65 compiles a component with a warning on purpose; both are async: true, so the capture picks up the other test's warning. Seen three times (two branches).
  3. Fixed mail receive timeout. test/mv_web/auth/password_reset_rate_limit_test.exs:149 waits 2 s for {:email, …}; the reset request now runs in the background, so under load the mail arrives later.
  4. assert_patch timeouts in member list tests. test/mv_web/live/member_live/index_add_filter_flow_test.exs:264-270 and earlier IndexFilterChipsTest / IndexValueControlTest (after their async switch): the patch arrives after the default timeout under load.
  5. Counting all rows of a table. test/mv_web/live/join_live_test.exs asserts count_join_requests() == count_before + n; a concurrent test's insert lands between the two reads.

Proposal

  • (1) Set ownership_timeout for Mv.Repo in config/test.exs to at least the largest test timeout, with a comment tying both values together; optionally make the property cheaper so it stays well below 120 s.
  • (2) Make the compile-warning tests independent of other tests' output: capture only what the compiled module produces (e.g. compile in an isolated process and collect its diagnostics) or move the deliberate-warning tests into one async: false module.
  • (3) Wait on a condition, not a fixed time: a longer assert_receive timeout tied to a test config value, or wait for the background task.
  • (4) Find the common cause of the late patches (e.g. a shared slow query or a missing render_async) and fix it there; raise timeouts only with a reason.
  • (5) Count only the rows the test created (filter by the test's own email or id).
  • Sweep for siblings of each class across the suite (same assertion shape) and fix them in the same change.

Acceptance criteria

  • Each of the five classes has a fix, and a sweep result names the other places of the same shape (fixed or verified safe).
  • A test that runs longer than 120 s but within its own timeout keeps its database connection.
  • just ci-dev passes 10 times in a row on a loaded host (two parallel suites), and just test-all 3 times, each with a recorded seed.
  • docs/ (testing guidelines) names the five patterns to avoid.

Dependencies

None

## Description Random failures block merges: the Renovate pull request #689 failed on CI in a test without a relation to the update, and every dependency update (e.g. `plug` 1.20.4) runs the same risk. Every developer and every pipeline run re-runs suites "until green", which hides real failures. The failures are not random. The evidence log (`flaky-tests.md`) shows five classes, each with a cause in the test setup: 1. **Sandbox ownership shorter than the test timeout.** `test/mv_web/member_export/scope_overview_parity_property_test.exs` allows 180 s (`@moduletag timeout: 180_000`); the sandbox keeps DBConnection's default `ownership_timeout` of 120 s. On a slow CI host the property loses its connection (`DBConnection.Holder.checkout … no process`). Seen on #689, Drone build 3096, seed 919006. 2. **Global stderr capture in async tests.** `test/mv_web/components/core_components_button_api_test.exs` (lines ~80–86) asserts an empty `capture_io(:stderr)` while `test/mv_web/components/feedback_callout_test.exs:65` compiles a component with a warning on purpose; both are `async: true`, so the capture picks up the other test's warning. Seen three times (two branches). 3. **Fixed mail receive timeout.** `test/mv_web/auth/password_reset_rate_limit_test.exs:149` waits 2 s for `{:email, …}`; the reset request now runs in the background, so under load the mail arrives later. 4. **`assert_patch` timeouts in member list tests.** `test/mv_web/live/member_live/index_add_filter_flow_test.exs:264-270` and earlier `IndexFilterChipsTest` / `IndexValueControlTest` (after their async switch): the patch arrives after the default timeout under load. 5. **Counting all rows of a table.** `test/mv_web/live/join_live_test.exs` asserts `count_join_requests() == count_before + n`; a concurrent test's insert lands between the two reads. ## Proposal - (1) Set `ownership_timeout` for `Mv.Repo` in `config/test.exs` to at least the largest test timeout, with a comment tying both values together; optionally make the property cheaper so it stays well below 120 s. - (2) Make the compile-warning tests independent of other tests' output: capture only what the compiled module produces (e.g. compile in an isolated process and collect its diagnostics) or move the deliberate-warning tests into one `async: false` module. - (3) Wait on a condition, not a fixed time: a longer `assert_receive` timeout tied to a test config value, or wait for the background task. - (4) Find the common cause of the late patches (e.g. a shared slow query or a missing `render_async`) and fix it there; raise timeouts only with a reason. - (5) Count only the rows the test created (filter by the test's own email or id). - Sweep for siblings of each class across the suite (same assertion shape) and fix them in the same change. ## Acceptance criteria - [ ] Each of the five classes has a fix, and a sweep result names the other places of the same shape (fixed or verified safe). - [ ] A test that runs longer than 120 s but within its own timeout keeps its database connection. - [ ] `just ci-dev` passes 10 times in a row on a loaded host (two parallel suites), and `just test-all` 3 times, each with a recorded seed. - [ ] `docs/` (testing guidelines) names the five patterns to avoid. ## Dependencies None
simon closed this issue 2026-11-20 09:01:18 +01:00
Sign in to join this conversation.
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference
local-it/mitgliederverwaltung#695
No description provided.