Make tests fail when the behaviour they name breaks closes #609

simon commented

2026-10-20 14:14:53 +02:00

Owner

closes #609

Description of the implemented changes

The changes were:

Bugfixing
New Feature
Breaking Change
Refactoring

Make tests fail when the behaviour they name breaks: many tests matched flashes, errors and page content with substring or whole-page checks that kept passing when the named behaviour broke. This PR adds exact-match test assertions (flash and field errors, page titles, single elements, list items, forbidden/not-found/invalid Ash errors, access-denied redirects, query counts), makes them the rule in CODE_GUIDELINES §4, hardens the tests across the domain and web layers, and drops tests whose input and checks a stricter sibling already covers (2640 → 2499 tests). Hardening surfaced two real bugs, fixed here with their regression tests.

What has been changed?

Bump ash_postgres past the retired 2.14.5 release
Add assertions that fail when the named behaviour breaks
Pin exact permission results and read deleted records with an allowed actor
Assert exact join request, fee cycle, import and validation results
Pin authentication, identity and system actor errors and integration payloads
Show the success and error messages of the membership fees tab
Name the HTTP status when the API error has an empty body
Check overview and show page results by row, cell and element
Check group members by id and pin group titles, redirects and flashes
Pin flashes, errors and controls of user, role, auth, import and join pages by element
Share the exact-text filter, access-denied check and query counter across tests
Drop tests that stricter siblings cover and build expected messages from gettext
Build expected strings and selectors by interpolation instead of <>
Build the Vereinfacht fallback error and the new-cycle period label by interpolation

Definition of Done

Code Quality

No new technical depths
Linting passed
Documentation is added were needed

Accessibility

New elements are properly defined with html-tags
Colour contrast follows WCAG criteria
Aria labels are added when needed
Everything is accessible by keyboard
Tab-Order is comprehensible
All interactive elements have a visible focus

Testing

Tests for new code are written
All tests pass
axe-core dev tools show no critical or major issues

Additional Notes

The ash_postgres bump (2.14.5 → 2.15.1, pulling Ash 3.36.1) is included because 2.14.5 was retired upstream and mix hex.audit fails on main without it; Mila's two bulk_destroy calls use no limit, so the retirement reason (limited bulk deletes on small tables) most likely did not affect it.
The last commit is a behaviour-neutral style change in lib/ (interpolation instead of <>), outside the otherwise test-only scope.
The review loop hit its cap of 4 iterations; the remaining findings were accepted.
Large commits: most test commits touch more than 5 files or 150 lines; the largest (user, role, auth, import and join pages) changes about 3200 lines over 36 files.
Pre-existing on main, not touched here: mix ash.codegen --check reports pending snapshot changes (ash_functions extension version, stale property snapshots).

closes #609 ## Description of the implemented changes The changes were: - [x] Bugfixing - [ ] New Feature - [ ] Breaking Change - [ ] Refactoring Make tests fail when the behaviour they name breaks: many tests matched flashes, errors and page content with substring or whole-page checks that kept passing when the named behaviour broke. This PR adds exact-match test assertions (flash and field errors, page titles, single elements, list items, forbidden/not-found/invalid Ash errors, access-denied redirects, query counts), makes them the rule in CODE_GUIDELINES §4, hardens the tests across the domain and web layers, and drops tests whose input and checks a stricter sibling already covers (2640 → 2499 tests). Hardening surfaced two real bugs, fixed here with their regression tests. ## What has been changed? - Bump ash_postgres past the retired 2.14.5 release - Add assertions that fail when the named behaviour breaks - Pin exact permission results and read deleted records with an allowed actor - Assert exact join request, fee cycle, import and validation results - Pin authentication, identity and system actor errors and integration payloads - Show the success and error messages of the membership fees tab - Name the HTTP status when the API error has an empty body - Check overview and show page results by row, cell and element - Check group members by id and pin group titles, redirects and flashes - Pin flashes, errors and controls of user, role, auth, import and join pages by element - Share the exact-text filter, access-denied check and query counter across tests - Drop tests that stricter siblings cover and build expected messages from gettext - Build expected strings and selectors by interpolation instead of <> - Build the Vereinfacht fallback error and the new-cycle period label by interpolation ## Definition of Done ### Code Quality - [ ] No new technical depths - [x] Linting passed - [x] Documentation is added were needed ### Accessibility - [ ] New elements are properly defined with html-tags - [ ] Colour contrast follows WCAG criteria - [ ] Aria labels are added when needed - [ ] Everything is accessible by keyboard - [ ] Tab-Order is comprehensible - [ ] All interactive elements have a visible focus ### Testing - [x] Tests for new code are written - [x] All tests pass - [ ] axe-core dev tools show no critical or major issues ## Additional Notes - The ash_postgres bump (2.14.5 → 2.15.1, pulling Ash 3.36.1) is included because 2.14.5 was retired upstream and `mix hex.audit` fails on main without it; Mila's two `bulk_destroy` calls use no limit, so the retirement reason (limited bulk deletes on small tables) most likely did not affect it. - The last commit is a behaviour-neutral style change in lib/ (interpolation instead of `<>`), outside the otherwise test-only scope. - The review loop hit its cap of 4 iterations; the remaining findings were accepted. - Large commits: most test commits touch more than 5 files or 150 lines; the largest (user, role, auth, import and join pages) changes about 3200 lines over 36 files. - Pre-existing on main, not touched here: `mix ash.codegen --check` reports pending snapshot changes (ash_functions extension version, stale property snapshots).

simon added 15 commits

2026-10-20 14:14:54 +02:00

chore(deps): bump ash_postgres past the retired 2.14.5 release 8050ae11b4

test(support): add assertions that fail when the named behaviour breaks 5bd6653095

test(authorization): pin exact permission results and read deleted records with an allowed actor cfb895c1a7

test(membership): assert exact join request, fee cycle, import and validation results cd75ebc696

test(accounts): pin authentication, identity and system actor errors and integration payloads c24e89a94c

fix(member): show the success and error messages of the membership fees tab 16694ba261

fix(vereinfacht): name the HTTP status when the API error has an empty body 046a8c6549

test(member): check overview and show page results by row, cell and element c014a595b9

test(group): check group members by id and pin group titles, redirects and flashes 52cb4119b1

test(web): pin flashes, errors and controls of user, role, auth, import and join pages by element a6bd573d16