Tests: remove the known causes of random red CI runs #695
Labels
No labels
bug
duplicate
enhancement
help wanted
high priority
invalid
L
low priority
M
medium priority
needs refinement
optional
question
S
security
technical improvement
UX Improvement
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set
Reference
local-it/mitgliederverwaltung#695
Loading…
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Description
Random failures block merges: the Renovate pull request #689 failed on CI in a test without a relation to the update, and every dependency update (e.g.
plug1.20.4) runs the same risk. Every developer and every pipeline run re-runs suites "until green", which hides real failures.The failures are not random. The evidence log (
flaky-tests.md) shows five classes, each with a cause in the test setup:test/mv_web/member_export/scope_overview_parity_property_test.exsallows 180 s (@moduletag timeout: 180_000); the sandbox keeps DBConnection's defaultownership_timeoutof 120 s. On a slow CI host the property loses its connection (DBConnection.Holder.checkout … no process). Seen on #689, Drone build 3096, seed 919006.test/mv_web/components/core_components_button_api_test.exs(lines ~80–86) asserts an emptycapture_io(:stderr)whiletest/mv_web/components/feedback_callout_test.exs:65compiles a component with a warning on purpose; both areasync: true, so the capture picks up the other test's warning. Seen three times (two branches).test/mv_web/auth/password_reset_rate_limit_test.exs:149waits 2 s for{:email, …}; the reset request now runs in the background, so under load the mail arrives later.assert_patchtimeouts in member list tests.test/mv_web/live/member_live/index_add_filter_flow_test.exs:264-270and earlierIndexFilterChipsTest/IndexValueControlTest(after their async switch): the patch arrives after the default timeout under load.test/mv_web/live/join_live_test.exsassertscount_join_requests() == count_before + n; a concurrent test's insert lands between the two reads.Proposal
ownership_timeoutforMv.Repoinconfig/test.exsto at least the largest test timeout, with a comment tying both values together; optionally make the property cheaper so it stays well below 120 s.async: falsemodule.assert_receivetimeout tied to a test config value, or wait for the background task.render_async) and fix it there; raise timeouts only with a reason.Acceptance criteria
just ci-devpasses 10 times in a row on a loaded host (two parallel suites), andjust test-all3 times, each with a recorded seed.docs/(testing guidelines) names the five patterns to avoid.Dependencies
None