Fix remaining flaky tests (unordered search result, async log capture, delete modal) #598

New issue

Closed

opened 2026-10-16 22:34:33 +02:00 by simon · 0 comments

simon commented

2026-10-16 22:34:33 +02:00

Owner

Motivation

#533 removed the structural cause of the earlier flakiness (FK deadlocks in create_member). Three
other flaky tests remain. They fail only in full just ci-dev / just check runs, never in isolation,
and on branches that do not touch the tested code. Each occurrence costs a full gate re-run
(about 6 minutes); finalizing #539 needed three full runs.

Observed failures

Run	Seed	Test	Symptom
#535 tdd	511498	`Mv.Membership.FuzzySearchTest` "returns only John Doe by fuzzy query 'john'"	result order flipped
#539 finalize	213477	same	result order flipped
#539 finalize	435950	`Mv.Authorization.Checks.HasPermissionTest` "logs authorization failure for missing role"	`assert log =~ "Authorization failed" or log == ""` fails
#593 finalize	—	`has_permission_test.exs` log-capture test	failed, passed on retry
#539 tdd	237	`MvWeb.CustomFieldLive.DeletionTest` "confirm deletion successfully deletes custom field with correct slug"	delete modal not found

All re-runs in isolation (including with the reported seed) were green.

Analysis so far

1. Order assertion on an unordered read — FuzzySearchTest

test/mv/membership/fuzzy_search_test.exs:51 asserts the exact order [john.id, alice.id].
The :search read action (Mv.Membership.Member, preparation Member.Preparations.FuzzySearch,
filter in Member.Search) has no sort, so Postgres returns plan/heap order, which depends on table state.
The file is async: false; this is order-dependence on DB state, not concurrency.
Open decision: should fuzzy_search/2 return results by relevance?
- yes → sort by similarity with a deterministic tie-breaker (e.g. id) and keep the ordered assertion;
- no → compare as a set in the test.
Check the same assertion shape in test/mv_web/member_export/scope_test.exs:52 and
test/mv_web/member_export/scope_overview_parity_property_test.exs:203 (fine if their code path sorts explicitly).

2. capture_log in an async test — HasPermissionTest

The file is async: true. The "Authorization failed" message is logged at debug
(lib/mv/authorization/checks/has_permission.ex:374) while config/test.exs sets level: :warning,
so normally nothing is captured and log == "" passes.
Hypothesis: capture_log also captures messages from other processes. Concurrent async tests emit
[warning] lines (OIDC authentication failures, join-email failures — visible in every CI log). If one
lands in the capture window, log is non-empty without "Authorization failed" and the test fails.
The assertion checks almost nothing (=~ X or == ""). Options: async: false, assert on the return
value only, or capture at :debug and match the own message explicitly.

3. LiveView modal not found — CustomFieldLive.DeletionTest

async: true, helper open_delete_modal/2. One occurrence so far; likely timing. Collect more
occurrences before changing anything.

Scope

Confirm the hypothesis for 2 (reproduce with a parallel Logger.warning during the capture), then fix.
Decide the relevance-order question for 1, fix the test (and the action if ordering is wanted); sweep the two sibling assertions.
Investigate 3 if it recurs; otherwise leave it documented.

Acceptance criteria

FuzzySearchTest and HasPermissionTest pass deterministically across 10 distinct seeded full-suite runs.
The relevance-order decision is recorded (moduledoc of Member.Search or the test).
just ci-dev green.

Notes

On every new flake: keep the full log, read _build/test/lib/mv/.mix_test_failures before re-running,
and add the seed and test name to this issue.

## Motivation #533 removed the structural cause of the earlier flakiness (FK deadlocks in `create_member`). Three other flaky tests remain. They fail only in full `just ci-dev` / `just check` runs, never in isolation, and on branches that do not touch the tested code. Each occurrence costs a full gate re-run (about 6 minutes); finalizing #539 needed three full runs. ## Observed failures | Run | Seed | Test | Symptom | |---|---|---|---| | #535 tdd | 511498 | `Mv.Membership.FuzzySearchTest` "returns only John Doe by fuzzy query 'john'" | result order flipped | | #539 finalize | 213477 | same | result order flipped | | #539 finalize | 435950 | `Mv.Authorization.Checks.HasPermissionTest` "logs authorization failure for missing role" | `assert log =~ "Authorization failed" or log == ""` fails | | #593 finalize | — | `has_permission_test.exs` log-capture test | failed, passed on retry | | #539 tdd | 237 | `MvWeb.CustomFieldLive.DeletionTest` "confirm deletion successfully deletes custom field with correct slug" | delete modal not found | All re-runs in isolation (including with the reported seed) were green. ## Analysis so far **1. Order assertion on an unordered read — `FuzzySearchTest`** - `test/mv/membership/fuzzy_search_test.exs:51` asserts the exact order `[john.id, alice.id]`. - The `:search` read action (`Mv.Membership.Member`, preparation `Member.Preparations.FuzzySearch`, filter in `Member.Search`) has no sort, so Postgres returns plan/heap order, which depends on table state. The file is `async: false`; this is order-dependence on DB state, not concurrency. - Open decision: should `fuzzy_search/2` return results by relevance? - yes → sort by similarity with a deterministic tie-breaker (e.g. id) and keep the ordered assertion; - no → compare as a set in the test. - Check the same assertion shape in `test/mv_web/member_export/scope_test.exs:52` and `test/mv_web/member_export/scope_overview_parity_property_test.exs:203` (fine if their code path sorts explicitly). **2. `capture_log` in an async test — `HasPermissionTest`** - The file is `async: true`. The "Authorization failed" message is logged at `debug` (`lib/mv/authorization/checks/has_permission.ex:374`) while `config/test.exs` sets `level: :warning`, so normally nothing is captured and `log == ""` passes. - Hypothesis: `capture_log` also captures messages from other processes. Concurrent async tests emit `[warning]` lines (OIDC authentication failures, join-email failures — visible in every CI log). If one lands in the capture window, `log` is non-empty without "Authorization failed" and the test fails. - The assertion checks almost nothing (`=~ X or == ""`). Options: `async: false`, assert on the return value only, or capture at `:debug` and match the own message explicitly. **3. LiveView modal not found — `CustomFieldLive.DeletionTest`** - `async: true`, helper `open_delete_modal/2`. One occurrence so far; likely timing. Collect more occurrences before changing anything. ## Scope 1. Confirm the hypothesis for 2 (reproduce with a parallel `Logger.warning` during the capture), then fix. 2. Decide the relevance-order question for 1, fix the test (and the action if ordering is wanted); sweep the two sibling assertions. 3. Investigate 3 if it recurs; otherwise leave it documented. ## Acceptance criteria - `FuzzySearchTest` and `HasPermissionTest` pass deterministically across 10 distinct seeded full-suite runs. - The relevance-order decision is recorded (moduledoc of `Member.Search` or the test). - `just ci-dev` green. ## Notes On every new flake: keep the full log, read `_build/test/lib/mv/.mix_test_failures` before re-running, and add the seed and test name to this issue.

simon added this to the Code and Test Refactoring | TI I milestone

2026-10-16 22:34:33 +02:00

simon added this to the Sprint 18: Oktober 2026 project

2026-10-16 22:34:33 +02:00

simon referenced this issue from a pull request that will close it,

2026-10-17 08:44:08 +02:00

Fix remaining flaky tests closes #598 #603

simon closed this issue

2026-10-17 08:58:45 +02:00

simon referenced this issue from a commit

2026-10-17 08:58:45 +02:00

Merge pull request 'Fix remaining flaky tests closes #598' (#603) from issue/mitgliederverwaltung-598 into main