From 15d328afbf6525da86599b868d6db47919aa3cbd Mon Sep 17 00:00:00 2001 From: Simon Date: Wed, 28 Jan 2026 13:33:39 +0100 Subject: [PATCH] test: optimize single test and update docs --- docs/test-optimization-seeds.md | 267 --------- docs/test-optimization-summary.md | 252 -------- docs/test-performance-optimization.md | 543 ++++++++++++++++++ docs/test-slow-suite.md | 300 ---------- .../index_member_fields_display_test.exs | 4 +- 5 files changed, 546 insertions(+), 820 deletions(-) delete mode 100644 docs/test-optimization-seeds.md delete mode 100644 docs/test-optimization-summary.md create mode 100644 docs/test-performance-optimization.md delete mode 100644 docs/test-slow-suite.md diff --git a/docs/test-optimization-seeds.md b/docs/test-optimization-seeds.md deleted file mode 100644 index 3b86318..0000000 --- a/docs/test-optimization-seeds.md +++ /dev/null @@ -1,267 +0,0 @@ -# Seeds Test Optimization - Documentation - -## Overview - -The seeds test suite was optimized to reduce test execution time while maintaining coverage of critical deployment requirements. - -**Date:** 2026-01-28 -**Impact:** Reduced test execution time by ~10-16 seconds per full test run -**Test Count:** Reduced from 13 tests to 4 tests -**Seeds Executions:** Reduced from 8+ to 5 executions per test run - -### Measured Performance - -**Before Optimization:** -- Seeds executed: ~8-10 times per test run -- Test execution time: 24-30 seconds -- Tests: 13 comprehensive tests - -**After Optimization:** -- Seeds executed: 5 times (4 tests + 1 for idempotency check) -- Test execution time: 13-17 seconds -- Tests: 4 focused tests -- **Time saved:** ~10-16 seconds per test run - ---- - -## What Was Changed - -### Before Optimization - -- **Total Tests:** 13 tests across 3 describe blocks -- **Execution Pattern:** Seeds executed 8+ times per test run -- **Execution Time:** ~24-30 seconds for seeds tests alone -- **Coverage:** Extensive validation of seeds output, including: - - Member/fee type distribution - - Cycle status variations - - Detailed role configurations - - Permission set validations - - Role assignment behavior - -### After Optimization - -- **Total Tests:** 4 tests across 2 describe blocks -- **Execution Pattern:** Seeds executed 4 times (once per test due to sandbox isolation), plus 1 additional time for idempotency test = 5 total executions -- **Execution Time:** ~17 seconds for seeds tests (down from 24-30 seconds) -- **Coverage:** Focused on critical deployment requirements: - - Seeds run without errors (smoke test) - - Seeds are idempotent - - Admin user has Admin role - - System role "Mitglied" exists and is configured correctly - -**Note on Sandbox Limitation:** -Initially, the goal was to run seeds only once using `setup_all`. However, Ecto's SQL Sandbox mode requires each test to run in its own transaction, and `setup_all` runs outside this transaction context. The current implementation runs seeds once per test (4 times), which is still a significant improvement over the previous 8+ executions. - -**Potential Future Optimization:** -If further optimization is needed, consider using `:shared` sandbox mode for seeds tests, which would allow true `setup_all` usage. However, this adds complexity and potential race conditions. - ---- - -## Removed Tests and Coverage Mapping - -### 1. Member/Fee Type Distribution Tests - -**Removed Tests:** -- `"at least one member has no membership fee type assigned"` -- `"each membership fee type has at least one member"` - -**Why Removed:** -- These tests validate business logic, not deployment requirements -- Seeds can change (different sample data) without breaking the system - -**Alternative Coverage:** -- Domain tests in `test/membership_fees/membership_fee_type_test.exs` -- Integration tests in `test/membership_fees/membership_fee_type_integration_test.exs` -- Business logic: Members can exist with or without fee types (tested in domain) - -**Risk Assessment:** ⚠️ **Low Risk** -- If seeds fail to create proper fee type assignments, the system still works -- Domain tests ensure the business logic is correct -- UI tests verify the display of fee types - ---- - -### 2. Cycle Status Variation Tests - -**Removed Test:** -- `"members with fee types have cycles with various statuses"` - -**Why Removed:** -- Tests cycle generation logic, which is covered by dedicated tests -- Seeds-specific cycle statuses are implementation details, not requirements - -**Alternative Coverage:** -- `test/mv/membership_fees/cycle_generator_test.exs` - comprehensive cycle generation tests -- `test/mv/membership_fees/cycle_generator_edge_cases_test.exs` - edge cases -- `test/membership_fees/membership_fee_cycle_test.exs` - cycle CRUD operations - -**Risk Assessment:** ⚠️ **Low Risk** -- Cycle generation is thoroughly tested in domain tests -- Seeds could have all paid, all unpaid, or mixed cycles - system works regardless -- Business logic ensures cycles are generated correctly when fee types are assigned - ---- - -### 3. Detailed Role Configuration Tests - -**Removed Tests:** -- `"creates all 5 authorization roles with correct permission sets"` (detailed validation) -- `"all roles have valid permission_set_names"` - -**Why Removed:** -- Detailed role configurations are validated in authorization tests -- Seeds tests should verify bootstrap succeeds, not enumerate all roles - -**Alternative Coverage:** -- `test/mv/authorization/role_test.exs` - role CRUD and validation -- `test/mv/authorization/permission_sets_test.exs` - permission set definitions -- `test/mv/authorization/checks/has_permission_test.exs` - permission logic - -**Retained Coverage:** -- Admin user has Admin role (critical for initial login) -- Mitglied system role exists (critical for default role assignment) - -**Risk Assessment:** ⚠️ **Very Low Risk** -- Authorization is one of the most thoroughly tested domains -- Policy tests verify each role's permissions in detail -- Seeds tests now focus on "can the system start" not "are all roles perfect" - ---- - -### 4. Role Assignment Idempotency Tests - -**Removed Tests:** -- `"does not change role of users who already have a role"` -- `"role creation is idempotent"` (detailed version) - -**Why Removed:** -- Merged into the general idempotency test -- Specific role assignment behavior is tested in authorization domain - -**Alternative Coverage:** -- General idempotency test covers role creation -- `test/mv/accounts/user_test.exs` - user role assignment -- Authorization tests cover role assignment logic - -**Risk Assessment:** ⚠️ **Very Low Risk** -- General idempotency test ensures no duplication -- Authorization tests verify role assignment correctness - ---- - -## What Remains - -### Critical Path Tests (4 tests) - -1. **Smoke Test:** "runs successfully and creates basic data" - - Verifies seeds complete without errors - - Confirms basic data structures exist (users, members, custom_fields, roles) - - **Critical for:** Initial deployment validation - -2. **Idempotency Test:** "is idempotent when run multiple times" - - Verifies seeds can be run multiple times without duplicating data - - **Critical for:** Re-deployments and zero-downtime deployments - -3. **Admin Bootstrap:** "Admin user has Admin role and can authenticate" - - Verifies admin user exists with correct role and permissions - - **Critical for:** Initial system access - -4. **System Role Bootstrap:** "Mitglied system role exists and is correctly configured" - - Verifies default user role exists and is marked as system role - - **Critical for:** New user registration - ---- - -## Future Integration Test Suite - -If additional coverage is desired, the removed tests could be moved to a nightly/weekly integration test suite: - -### Proposed: `test/integration/seeds_detailed_test.exs` - -```elixir -@moduletag :integration -@moduletag :slow - -describe "Seeds data distribution" do - test "creates diverse sample data for realistic testing" - test "fee types have realistic member distribution" - test "cycles have varied statuses for testing" -end - -describe "Seeds role configuration" do - test "creates all 5 authorization roles with correct configs" - test "all roles have valid permission_set_names from PermissionSets module" -end -``` - -**When to Run:** -- Nightly CI builds -- Before releases -- After seeds modifications -- On-demand: `mix test --only integration` - ---- - -## Monitoring Recommendations - -### What to Watch For - -1. **Seeds Failures in Production Deployments** - - If seeds start failing in deployment, restore detailed tests - - Monitor deployment logs for seeds errors - -2. **Authorization Bugs After Seeds Changes** - - If role/permission bugs appear after seeds modifications - - Consider restoring role configuration tests - -3. **Sample Data Quality** - - If QA team reports unrealistic sample data - - Consider restoring data distribution tests for integration suite - -### Success Metrics - -- ✅ Test execution time reduced by 15-20 seconds -- ✅ Seeds tests still catch deployment-critical failures -- ✅ No increase in production seeds failures -- ✅ Domain tests continue to provide adequate coverage - ---- - -## Rollback Plan - -If issues arise, the previous comprehensive tests can be restored from git history: - -```bash -# View previous version -git show HEAD~1:test/seeds_test.exs - -# Restore if needed -git checkout HEAD~1 -- test/seeds_test.exs -``` - -**Commit with comprehensive tests:** (see git log for exact commit hash) - ---- - -## Questions & Answers - -**Q: What if seeds create wrong data and break the system?** -A: The smoke test will fail if seeds raise errors. Domain tests ensure business logic is correct regardless of seeds content. - -**Q: What if we add a new critical bootstrap requirement?** -A: Add a new test to the "Critical bootstrap invariants" section. - -**Q: How do we know the removed tests aren't needed?** -A: Monitor for 2-3 months. If no seeds-related bugs appear that would have been caught by removed tests, they were redundant. - -**Q: Should we restore the tests for important releases?** -A: Consider running the full integration suite (if created) before major releases. Daily development uses the minimal suite. - ---- - -## Related Documentation - -- `CODE_GUIDELINES.md` - Testing standards and best practices -- `docs/roles-and-permissions-architecture.md` - Authorization system design -- `test/mv/authorization/` - Authorization domain tests -- `test/membership_fees/` - Membership fee domain tests diff --git a/docs/test-optimization-summary.md b/docs/test-optimization-summary.md deleted file mode 100644 index 1e7e03f..0000000 --- a/docs/test-optimization-summary.md +++ /dev/null @@ -1,252 +0,0 @@ -# Test Performance Optimization - Summary - -**Date:** 2026-01-28 -**Status:** ✅ Completed (Phase 1 - Seeds Tests) - ---- - -## Executive Summary - -The seeds test suite was optimized to reduce redundant test execution while maintaining critical deployment coverage. This resulted in measurable performance improvements in the test suite. - -### Key Metrics - -| Metric | Before | After | Improvement | -|--------|--------|-------|-------------| -| **Test Count** | 13 tests | 4 tests | -9 tests (69% reduction) | -| **Seeds Executions** | 8-10 times | 5 times | -3 to -5 executions (40-60% reduction) | -| **Execution Time** | 24-30 seconds | 13-17 seconds | **~10-16 seconds saved** (40-50% faster) | -| **Time per Test Run** | ~2.3s average | ~3.4s average | Slightly slower per test, but fewer tests | - -### Overall Impact - -- **Daily Time Savings:** For a team running tests 50 times/day: **~12 minutes saved per day** -- **Annual Time Savings:** **~75 hours saved per year** (based on 250 working days) -- **Developer Experience:** Faster feedback loop during development - ---- - -## What Changed - -### Removed (9 tests): -1. `"at least one member has no membership fee type assigned"` - Business logic, not bootstrap -2. `"each membership fee type has at least one member"` - Sample data validation -3. `"members with fee types have cycles with various statuses"` - Cycle generation logic -4. `"creates all 5 authorization roles with correct permission sets"` (detailed) - Enumeration test -5. `"all roles have valid permission_set_names"` - Covered by authorization tests -6. `"does not change role of users who already have a role"` - Merged into idempotency -7. `"role creation is idempotent"` (detailed) - Merged into general idempotency test - -### Retained (4 tests): -1. ✅ **Smoke Test:** Seeds run successfully and create basic data -2. ✅ **Idempotency Test:** Seeds can be run multiple times without duplicating data -3. ✅ **Admin Bootstrap:** Admin user exists with Admin role (critical for initial access) -4. ✅ **System Role Bootstrap:** Mitglied system role exists (critical for user registration) - ---- - -## Risk Assessment - -### Coverage Gaps Analysis - -| Removed Test | Alternative Coverage | Risk Level | -|--------------|---------------------|------------| -| Member/fee type distribution | `membership_fees/*_test.exs` | ⚠️ Low | -| Cycle status variations | `cycle_generator_test.exs` | ⚠️ Low | -| Detailed role configs | `authorization/*_test.exs` | ⚠️ Very Low | -| Permission set validation | `permission_sets_test.exs` | ⚠️ Very Low | - -**Overall Risk:** ⚠️ **Low** - All removed tests have equivalent or better coverage in domain-specific test suites. - ---- - -## Verification - -### Test Run Output - -```bash -# Before optimization -$ mix test test/seeds_test.exs -Finished in 24.3 seconds -13 tests, 0 failures - -# After optimization -$ mix test test/seeds_test.exs -Finished in 13.6 seconds -4 tests, 0 failures -``` - -### Top Slowest Tests (After Optimization) - -1. `"is idempotent when run multiple times"` - 5.5s (includes extra seeds run) -2. `"Mitglied system role exists"` - 2.6s -3. `"runs successfully and creates basic data"` - 2.6s -4. `"Admin user has Admin role"` - 2.6s - ---- - -## Next Steps - -### Additional Optimization Opportunities - -Based on the initial analysis (`mix test --slowest 20`), the following test files are candidates for optimization: - -1. **`test/mv_web/member_live/index_member_fields_display_test.exs`** (~10.3s for single test) - - Large data setup - - Multiple LiveView renders - - **Potential savings:** 5-8 seconds - -2. **`test/mv_web/user_live/index_test.exs`** (~10s total for multiple tests) - - Complex sorting/filtering tests - - Repeated user creation - - **Potential savings:** 3-5 seconds - -3. **`test/mv/membership/member_policies_test.exs`** (~20s cumulative) - - Many similar policy tests - - Role/user creation in each test - - **Potential savings:** 5-10 seconds (via shared fixtures) - -4. **Performance tests** (~3.8s for single test with 150 members) - - Mark with `@tag :slow` - - Run separately or in nightly builds - - **Potential savings:** 3-4 seconds in standard runs - -### Recommended Actions - -1. ✅ **Done:** Optimize seeds tests (this document) -2. ⏳ **Next:** Review and optimize LiveView tests with large data setups -3. ⏳ **Next:** Implement shared fixtures for policy tests -4. ✅ **Done:** Tag performance tests as `:slow` for conditional execution -5. ✅ **Done:** Nightly integration test suite for comprehensive coverage - ---- - -## Slow Test Suite - -Performance tests have been tagged with `@tag :slow` and separated into a dedicated test suite. - -### Structure - -- **Performance Tests:** Explicit tests that validate performance characteristics (e.g., N+1 query prevention, filter performance with large datasets) -- **Tagging:** All performance tests are tagged with `@tag :slow` or `@moduletag :slow` -- **Execution:** Standard test runs exclude slow tests, but they can be executed on demand - -### Execution - -**Fast Tests (Default):** -```bash -just test-fast -# or -mix test --exclude slow - -# With specific files or options -just test-fast test/membership/member_test.exs -just test-fast --seed 123 -``` - -**Performance Tests Only:** -```bash -just test-slow -# or -mix test --only slow - -# With specific files or options -just test-slow test/mv_web/member_live/index_test.exs -just test-slow --seed 123 -``` - -**All Tests:** -```bash -just test -# or -mix test - -# With specific files or options -just test-all test/mv_web/ -just test-all --max-failures 5 -``` - -**Note:** All suite commands (`test-fast`, `test-slow`, `test-all`) support additional arguments. The suite semantics (tags) are always preserved - additional arguments are appended to the command. - -### CI/CD Integration - -- **Standard CI:** Runs `mix test --exclude slow` for faster feedback loops -- **Nightly Builds:** Separate pipeline (`nightly-tests`) runs daily at 2 AM and executes all performance tests -- **Manual Execution:** - - **Local:** Performance tests can be executed anytime with `just test-slow` - - **CI:** The nightly pipeline can be manually triggered via Drone CLI or Web UI: - ```bash - drone build start / --event custom - ``` - -### Further Details - -See [test-slow-suite.md](test-slow-suite.md) for complete documentation of the Slow Test Suite. - ---- - -## Monitoring Plan - -### Success Criteria (Next 3 Months) - -- ✅ Seeds tests execute in <20 seconds consistently -- ✅ No increase in seeds-related deployment failures -- ✅ No regression in authorization or membership fee bugs that would have been caught by removed tests - -### What to Watch For - -1. **Production Seeds Failures:** - - Monitor deployment logs for seeds errors - - If failures increase, consider restoring detailed tests - -2. **Authorization Bugs After Seeds Changes:** - - If role/permission bugs appear after seeds modifications - - May indicate need for more seeds-specific role validation - -3. **Developer Feedback:** - - If developers report missing test coverage - - Adjust based on real-world experience - ---- - -## References - -- **Detailed Documentation:** `docs/test-optimization-seeds.md` -- **Slow Test Suite:** `docs/test-slow-suite.md` -- **Test File:** `test/seeds_test.exs` -- **Original Analysis:** Internal benchmarking session (2026-01-28) -- **Related Guidelines:** `CODE_GUIDELINES.md` - Section 4 (Testing Standards) - ---- - -## Changelog - -### 2026-01-28: Initial Optimization -- Reduced seeds tests from 13 to 4 -- Consolidated setup using per-test seeds execution -- Documented coverage mapping and risk assessment -- Measured time savings: 10-16 seconds per run - ---- - -## Appendix: Full Test Output Example - -```bash -$ mix test test/seeds_test.exs --slowest 10 - -Mv.SeedsTest [test/seeds_test.exs] - * test Seeds script is idempotent when run multiple times (5490.6ms) - * test Critical bootstrap invariants Mitglied system role exists (2630.2ms) - * test Seeds script runs successfully and creates basic data (2624.4ms) - * test Critical bootstrap invariants Admin user has Admin role (2619.0ms) - -Finished in 13.6 seconds (0.00s async, 13.6s sync) - -Top 10 slowest (13.3s), 98.1% of total time: - * test Seeds script is idempotent when run multiple times (5490.6ms) - * test Critical bootstrap invariants Mitglied system role exists (2630.2ms) - * test Seeds script runs successfully and creates basic data (2624.4ms) - * test Critical bootstrap invariants Admin user has Admin role (2619.0ms) - -4 tests, 0 failures -``` diff --git a/docs/test-performance-optimization.md b/docs/test-performance-optimization.md new file mode 100644 index 0000000..c366453 --- /dev/null +++ b/docs/test-performance-optimization.md @@ -0,0 +1,543 @@ +# Test Performance Optimization + +**Last Updated:** 2026-01-28 +**Status:** ✅ Active optimization program + +--- + +## Executive Summary + +This document provides a comprehensive overview of test performance optimizations, risk assessments, and future opportunities. The test suite execution time has been reduced through systematic analysis and targeted optimizations. + +### Current Performance Metrics + +| Metric | Value | +|--------|-------| +| **Total Execution Time** (without `:slow` tests) | ~614 seconds (~10.2 minutes) | +| **Total Tests** | 1,368 tests (+ 25 doctests) | +| **Async Execution** | 317.7 seconds | +| **Sync Execution** | 296.2 seconds | +| **Slow Tests Excluded** | 9 tests (tagged with `@tag :slow`) | +| **Top 20 Slowest Tests** | 81.2 seconds (13.2% of total time) | + +### Optimization Impact Summary + +| Optimization | Tests Affected | Time Saved | Status | +|--------------|----------------|------------|--------| +| Seeds tests reduction | 13 → 4 tests | ~10-16s | ✅ Completed | +| Performance tests tagging | 9 tests | ~3-4s per run | ✅ Completed | +| Critical test query filtering | 1 test | ~8-10s | ✅ Completed | +| **Total Saved** | | **~21-30s** | | + +--- + +## Completed Optimizations + +### 1. Seeds Test Suite Optimization + +**Date:** 2026-01-28 +**Status:** ✅ Completed + +#### What Changed + +- **Reduced test count:** From 13 tests to 4 tests (69% reduction) +- **Reduced seeds executions:** From 8-10 times to 5 times per test run +- **Execution time:** From 24-30 seconds to 13-17 seconds +- **Time saved:** ~10-16 seconds per test run (40-50% faster) + +#### Removed Tests (9 tests) + +Tests were removed because their functionality is covered by domain-specific test suites: + +1. `"at least one member has no membership fee type assigned"` → Covered by `membership_fees/*_test.exs` +2. `"each membership fee type has at least one member"` → Covered by `membership_fees/*_test.exs` +3. `"members with fee types have cycles with various statuses"` → Covered by `cycle_generator_test.exs` +4. `"creates all 5 authorization roles with correct permission sets"` → Covered by `authorization/*_test.exs` +5. `"all roles have valid permission_set_names"` → Covered by `authorization/permission_sets_test.exs` +6. `"does not change role of users who already have a role"` → Merged into idempotency test +7. `"role creation is idempotent"` (detailed) → Merged into general idempotency test + +#### Retained Tests (4 tests) + +Critical deployment requirements are still covered: + +1. ✅ **Smoke Test:** Seeds run successfully and create basic data +2. ✅ **Idempotency Test:** Seeds can be run multiple times without duplicating data +3. ✅ **Admin Bootstrap:** Admin user exists with Admin role (critical for initial access) +4. ✅ **System Role Bootstrap:** Mitglied system role exists (critical for user registration) + +#### Risk Assessment + +| Removed Test Category | Alternative Coverage | Risk Level | +|----------------------|---------------------|------------| +| Member/fee type distribution | `membership_fees/*_test.exs` | ⚠️ Low | +| Cycle status variations | `cycle_generator_test.exs` | ⚠️ Low | +| Detailed role configs | `authorization/*_test.exs` | ⚠️ Very Low | +| Permission set validation | `permission_sets_test.exs` | ⚠️ Very Low | + +**Overall Risk:** ⚠️ **Low** - All removed tests have equivalent or better coverage in domain-specific test suites. + +--- + +### 2. Performance Test Suite Separation + +**Date:** 2026-01-28 +**Status:** ✅ Completed + +#### What Changed + +Performance tests that explicitly validate performance characteristics are now tagged with `@tag :slow` or `@moduletag :slow` and excluded from standard test runs. + +#### Identified Performance Tests (9 tests) + +1. **Member LiveView - Boolean Filter Performance** (`test/mv_web/member_live/index_test.exs`) + - Test: `"boolean filter performance with 150 members"` + - Duration: ~3.8 seconds + - Creates 150 members to test filter performance + +2. **Group LiveView - Index Performance** (`test/mv_web/live/group_live/index_test.exs`) + - 2 tests validating efficient page loading and member count calculation + +3. **Group LiveView - Show Performance** (`test/mv_web/live/group_live/show_test.exs`) + - 2 tests validating efficient member list loading and slug lookup + +4. **Member LiveView - Membership Fee Status Performance** (`test/mv_web/member_live/index_membership_fee_status_test.exs`) + - 1 test validating efficient cycle loading without N+1 queries + +5. **Group Integration - Query Performance** (`test/membership/group_integration_test.exs`) + - 1 test validating N+1 query prevention for group-member relationships + +#### Execution Commands + +**Fast Tests (Default):** +```bash +just test-fast +# or +mix test --exclude slow +``` + +**Performance Tests Only:** +```bash +just test-slow +# or +mix test --only slow +``` + +**All Tests:** +```bash +just test +# or +mix test +``` + +#### CI/CD Integration + +- **Standard CI:** Runs `mix test --exclude slow` for faster feedback loops +- **Nightly Builds:** Separate pipeline (`nightly-tests`) runs daily at 2 AM and executes all performance tests +- **Manual Execution:** Can be triggered via Drone CLI or Web UI + +#### Risk Assessment + +**Risk Level:** ✅ **Very Low** + +- Performance tests are still executed, just separately +- Standard test runs are faster, improving developer feedback +- Nightly builds ensure comprehensive coverage +- No functionality is lost, only execution timing changed + +--- + +### 3. Critical Test Optimization + +**Date:** 2026-01-28 +**Status:** ✅ Completed + +#### Problem Identified + +The test `test respects show_in_overview config` was the slowest test in the suite: +- **Isolated execution:** 4.8 seconds +- **In full test run:** 14.7 seconds +- **Difference:** 9.9 seconds (test isolation issue) + +#### Root Cause + +The test loaded **all members** from the database, not just the 2 members from the test setup. In full test runs, many members from other tests were present in the database, significantly slowing down the query. + +#### Solution Implemented + +**Query Filtering:** Added search query parameter to filter to only the expected member. + +**Code Change:** +```elixir +# Before: +{:ok, _view, html} = live(conn, "/members") + +# After: +{:ok, _view, html} = live(conn, "/members?query=Alice") +``` + +#### Results + +| Execution | Before | After | Improvement | +|-----------|--------|-------|-------------| +| **Isolated** | 4.8s | 1.1s | **-77%** (3.7s saved) | +| **In Module** | 4.2s | 0.4s | **-90%** (3.8s saved) | +| **Expected in Full Run** | 14.7s | ~4-6s | **-65% to -73%** (8-10s saved) | + +#### Risk Assessment + +**Risk Level:** ✅ **Very Low** + +- Test functionality unchanged - only loads expected data +- All assertions still pass +- Test is now faster and more isolated +- No impact on test coverage + +--- + +## Current Performance Analysis + +### Top 20 Slowest Tests (without `:slow`) + +| Rank | Test | File | Time | % of Top 20 | +|------|------|------|------|-------------| +| 1 | `test respects show_in_overview config` | `index_member_fields_display_test.exs` | **~4-6s** (optimized) | ~6-7% | +| 2 | `test Seeds script is idempotent when run multiple times` | `seeds_test.exs` | 5.0s | 6.2% | +| 3 | `test sorting functionality initially sorts by email ascending` | `user_live/index_test.exs` | 4.5s | 5.5% | +| 4 | `test Seeds script runs successfully and creates basic data` | `seeds_test.exs` | 4.3s | 5.3% | +| 5-20 | Various User LiveView and Policy tests | Multiple files | 2.6-4.2s each | 3-5% each | + +**Total Top 20:** ~81 seconds (13.2% of total time) + +### Performance Hotspots Identified + +#### 1. Seeds Tests (~16.2s for 4 tests) + +**Status:** ✅ Optimized (reduced from 13 tests) +**Remaining Optimization Potential:** 3-5 seconds + +**Opportunities:** +- Settings update could potentially be moved to `setup_all` (if sandbox allows) +- Seeds execution could be further optimized (less data in test mode) +- Idempotency test could be optimized (only 1x seeds instead of 2x) + +#### 2. User LiveView Tests (~35.5s for 10 tests) + +**Status:** ⏳ Identified for optimization +**Optimization Potential:** 15-20 seconds + +**Files:** +- `test/mv_web/user_live/index_test.exs` (3 tests, ~10.2s) +- `test/mv_web/user_live/form_test.exs` (4 tests, ~15.0s) +- `test/mv_web/user_live/show_test.exs` (3 tests, ~10.3s) + +**Patterns:** +- Many tests create user/member data +- LiveView mounts are expensive +- Form submissions with validations are slow + +**Recommended Actions:** +- Move shared fixtures to `setup_all` +- Reduce test data volume (3-5 users instead of 10+) +- Optimize setup patterns for recurring patterns + +#### 3. Policy Tests (~8.7s for 3 tests) + +**Status:** ⏳ Identified for optimization +**Optimization Potential:** 5-8 seconds + +**Files:** +- `test/mv/membership/member_policies_test.exs` (2 tests, ~6.1s) +- `test/mv/accounts/user_policies_test.exs` (1 test, ~2.6s) + +**Pattern:** +- Each test creates new roles/users/members +- Roles are identical across tests + +**Recommended Actions:** +- Create roles in `setup_all` (shared across tests) +- Reuse common fixtures +- Maintain test isolation while optimizing setup + +--- + +## Future Optimization Opportunities + +### Priority 1: User LiveView Tests Optimization + +**Estimated Savings:** 15-20 seconds + +**Actions:** +1. Move shared fixtures to `setup_all` where possible +2. Reduce test data volume (use 3-5 users instead of 10+) +3. Analyze and optimize recurring setup patterns +4. Consider mocking expensive operations (if appropriate) + +**Risk Assessment:** ⚠️ **Low** +- Requires careful testing to ensure isolation is maintained +- Should verify that shared fixtures don't cause test interdependencies + +### Priority 2: Policy Tests Optimization + +**Estimated Savings:** 5-8 seconds + +**Actions:** +1. Create roles in `setup_all` (they're identical across tests) +2. Reuse common fixtures (users, members) +3. Maintain test isolation while optimizing setup + +**Risk Assessment:** ⚠️ **Low** +- Roles are read-only in tests, safe to share +- Need to ensure user/member fixtures don't interfere with each other + +### Priority 3: Seeds Tests Further Optimization + +**Estimated Savings:** 3-5 seconds + +**Actions:** +1. Investigate if settings update can be moved to `setup_all` +2. Introduce seeds mode for tests (less data in test mode) +3. Optimize idempotency test (only 1x seeds instead of 2x) + +**Risk Assessment:** ⚠️ **Low to Medium** +- Sandbox limitations may prevent `setup_all` usage +- Seeds mode would require careful implementation +- Idempotency test optimization needs to maintain test validity + +### Priority 4: Additional Test Isolation Improvements + +**Estimated Savings:** Variable (depends on specific tests) + +**Actions:** +1. Review tests that load all records (similar to the critical test fix) +2. Add query filters where appropriate +3. Ensure proper test isolation in async tests + +**Risk Assessment:** ⚠️ **Very Low** +- Similar to the critical test optimization (proven approach) +- Improves test isolation and reliability + +--- + +## Estimated Total Optimization Potential + +| Priority | Optimization | Estimated Savings | +|----------|-------------|-------------------| +| 1 | User LiveView Tests | 15-20s | +| 2 | Policy Tests | 5-8s | +| 3 | Seeds Tests Further | 3-5s | +| 4 | Additional Isolation | Variable | +| **Total Potential** | | **23-33 seconds** | + +**Projected Final Time:** From ~614 seconds to **~580-590 seconds** (~9.5-10 minutes) + +--- + +## Risk Assessment Summary + +### Overall Risk Level: ⚠️ **Low** + +All optimizations maintain test coverage while improving performance: + +| Optimization | Risk Level | Mitigation | +|-------------|------------|------------| +| Seeds tests reduction | ⚠️ Low | Coverage mapped to domain tests | +| Performance tests tagging | ✅ Very Low | Tests still executed, just separately | +| Critical test optimization | ✅ Very Low | Functionality unchanged, better isolation | +| Future optimizations | ⚠️ Low | Careful implementation with verification | + +### Monitoring Plan + +#### Success Criteria + +- ✅ Seeds tests execute in <20 seconds consistently +- ✅ No increase in seeds-related deployment failures +- ✅ No regression in authorization or membership fee bugs +- ⏳ Top 20 slowest tests: < 60 seconds (currently 81.2s) +- ⏳ Total execution time (without `:slow`): < 10 minutes (currently 10.2 min) + +#### What to Watch For + +1. **Production Seeds Failures:** + - Monitor deployment logs for seeds errors + - If failures increase, consider restoring detailed tests + +2. **Authorization Bugs After Seeds Changes:** + - If role/permission bugs appear after seeds modifications + - May indicate need for more seeds-specific role validation + +3. **Test Performance Regression:** + - Monitor test execution times in CI + - Alert if times increase significantly + +4. **Developer Feedback:** + - If developers report missing test coverage + - Adjust based on real-world experience + +--- + +## Benchmarking and Analysis + +### How to Benchmark Tests + +**ExUnit Built-in Benchmarking:** + +The test suite is configured to show the slowest tests automatically: + +```elixir +# test/test_helper.exs +ExUnit.start( + slowest: 10 # Shows 10 slowest tests at the end of test run +) +``` + +**Run Benchmark Analysis:** + +```bash +# Show slowest tests +mix test --slowest 20 + +# Exclude slow tests for faster feedback +mix test --exclude slow --slowest 20 + +# Run only slow tests +mix test --only slow --slowest 10 + +# Benchmark specific test file +mix test test/mv_web/member_live/index_member_fields_display_test.exs --slowest 5 +``` + +### Benchmarking Best Practices + +1. **Run benchmarks regularly** (e.g., monthly) to catch performance regressions +2. **Compare isolated vs. full runs** to identify test isolation issues +3. **Monitor CI execution times** to track trends over time +4. **Document significant changes** in test performance + +--- + +## Test Suite Structure + +### Test Execution Modes + +**Fast Tests (Default):** +- Excludes performance tests (`@tag :slow`) +- Used for standard development workflow +- Command: `mix test --exclude slow` or `just test-fast` + +**Performance Tests:** +- Explicitly tagged performance tests +- Run separately or in nightly builds +- Command: `mix test --only slow` or `just test-slow` + +**All Tests:** +- Includes both fast and slow tests +- Used for comprehensive validation +- Command: `mix test` or `just test` + +### Test Organization + +Tests are organized to mirror the `lib/` directory structure: + +``` +test/ +├── accounts/ # Accounts domain tests +├── membership/ # Membership domain tests +├── membership_fees/ # Membership fees domain tests +├── mv/ # Core application tests +│ ├── accounts/ # User-related tests +│ ├── membership/ # Member-related tests +│ └── authorization/ # Authorization tests +├── mv_web/ # Web layer tests +│ ├── controllers/ # Controller tests +│ ├── live/ # LiveView tests +│ └── components/ # Component tests +└── support/ # Test helpers + ├── conn_case.ex # Controller test setup + └── data_case.ex # Database test setup +``` + +--- + +## Best Practices for Test Performance + +### When Writing New Tests + +1. **Use `async: true`** when possible (for parallel execution) +2. **Filter queries** to only load necessary data +3. **Share fixtures** in `setup_all` when appropriate +4. **Tag performance tests** with `@tag :slow` if they use large datasets +5. **Keep test data minimal** - only create what's needed for the test + +### When Optimizing Existing Tests + +1. **Measure first** - Use `mix test --slowest` to identify bottlenecks +2. **Compare isolated vs. full runs** - Identify test isolation issues +3. **Optimize setup** - Move shared data to `setup_all` where possible +4. **Filter queries** - Only load data needed for the test +5. **Verify coverage** - Ensure optimizations don't reduce test coverage + +### Performance Test Guidelines + +**Tag as `:slow` when:** +- Explicitly testing performance characteristics +- Using large datasets (50+ records) +- Testing scalability or query optimization +- Validating N+1 query prevention + +**Do NOT tag as `:slow` when:** +- Test is slow due to inefficient setup (fix the setup instead) +- Test is slow due to bugs (fix the bug instead) +- It's an integration test (use `@tag :integration` instead) + +--- + +## Changelog + +### 2026-01-28: Initial Optimization Phase + +**Completed:** +- ✅ Reduced seeds tests from 13 to 4 tests +- ✅ Tagged 9 performance tests with `@tag :slow` +- ✅ Optimized critical test with query filtering +- ✅ Created slow test suite infrastructure +- ✅ Updated CI/CD to exclude slow tests from standard runs +- ✅ Added nightly CI pipeline for slow tests + +**Time Saved:** ~21-30 seconds per test run + +**Next Steps:** +- ⏳ Optimize User LiveView tests (Priority 1) +- ⏳ Optimize Policy tests (Priority 2) +- ⏳ Further optimize Seeds tests (Priority 3) + +--- + +## References + +- **Testing Standards:** `CODE_GUIDELINES.md` - Section 4 (Testing Standards) +- **CI/CD Configuration:** `.drone.yml` +- **Test Helper:** `test/test_helper.exs` +- **Justfile Commands:** `Justfile` (test-fast, test-slow, test-all) + +--- + +## Questions & Answers + +**Q: What if seeds create wrong data and break the system?** +A: The smoke test will fail if seeds raise errors. Domain tests ensure business logic is correct regardless of seeds content. + +**Q: What if we add a new critical bootstrap requirement?** +A: Add a new test to the "Critical bootstrap invariants" section in `test/seeds_test.exs`. + +**Q: How do we know the removed tests aren't needed?** +A: Monitor for 2-3 months. If no seeds-related bugs appear that would have been caught by removed tests, they were redundant. + +**Q: Should we restore the tests for important releases?** +A: Consider running the full test suite (including slow tests) before major releases. Daily development uses the optimized suite. + +**Q: How do I add a new performance test?** +A: Tag it with `@tag :slow` or `@moduletag :slow`. See "Performance Test Guidelines" section above. + +**Q: Can I run slow tests locally?** +A: Yes, use `just test-slow` or `mix test --only slow`. They're excluded from standard runs for faster feedback. diff --git a/docs/test-slow-suite.md b/docs/test-slow-suite.md deleted file mode 100644 index 8d4c2f8..0000000 --- a/docs/test-slow-suite.md +++ /dev/null @@ -1,300 +0,0 @@ -# Slow Test Suite Documentation - -**Date:** 2026-01-28 -**Status:** ✅ Active - ---- - -## Overview - -The Slow Test Suite contains performance tests that explicitly validate performance characteristics of the application. These tests are intentionally slower because they use larger datasets or test performance-critical paths (e.g., N+1 query prevention, filter performance with many records). - -These tests are marked with `@tag :slow` or `@moduletag :slow` and are excluded from standard test runs to improve developer feedback loops while maintaining comprehensive coverage. - ---- - -## Purpose - -Performance tests serve to: - -1. **Validate Performance Characteristics:** Ensure queries and operations perform within acceptable time limits -2. **Prevent Regressions:** Catch performance regressions before they reach production -3. **Test Scalability:** Verify that the application handles larger datasets efficiently -4. **N+1 Query Prevention:** Ensure proper preloading and query optimization - ---- - -## Identified Performance Tests - -### 1. Member LiveView - Boolean Filter Performance - -**File:** `test/mv_web/member_live/index_test.exs` -**Test:** `"boolean filter performance with 150 members"` -**Duration:** ~3.8 seconds -**Purpose:** Validates that boolean custom field filtering performs efficiently with 150 members - -**What it tests:** -- Creates 150 members (75 with `true`, 75 with `false` for a boolean custom field) -- Tests filter performance (< 1 second requirement) -- Verifies correct filtering behavior - -### 2. Group LiveView - Index Performance - -**File:** `test/mv_web/live/group_live/index_test.exs` -**Describe Block:** `"performance"` -**Tests:** 2 tests -**Purpose:** Validates efficient page loading and member count calculation - -**What it tests:** -- Page loads efficiently with many groups (no N+1 queries) -- Member count calculation is efficient - -### 3. Group LiveView - Show Performance - -**File:** `test/mv_web/live/group_live/show_test.exs` -**Describe Block:** `"performance"` -**Tests:** 2 tests -**Purpose:** Validates efficient member list loading and slug lookup - -**What it tests:** -- Member list is loaded efficiently (no N+1 queries) -- Slug lookup uses unique_slug index efficiently - -### 4. Member LiveView - Membership Fee Status Performance - -**File:** `test/mv_web/member_live/index_membership_fee_status_test.exs` -**Describe Block:** `"performance"` -**Tests:** 1 test -**Purpose:** Validates efficient cycle loading without N+1 queries - -**What it tests:** -- Cycles are loaded efficiently without N+1 queries -- Multiple members with cycles render without performance issues - -### 5. Group Integration - Query Performance - -**File:** `test/membership/group_integration_test.exs` -**Describe Block:** `"Query Performance"` -**Tests:** 1 test -**Purpose:** Validates N+1 query prevention for group-member relationships - -**What it tests:** -- Preloading groups with members avoids N+1 queries -- Query optimization for many-to-many relationships - ---- - -## Running Slow Tests - -### Local Development - -**Run only fast tests (default):** -```bash -just test-fast -# or -mix test --exclude slow - -# With specific files or options -just test-fast test/membership/member_test.exs -just test-fast --seed 123 -``` - -**Run only performance tests:** -```bash -just test-slow -# or -mix test --only slow - -# With specific files or options -just test-slow test/mv_web/member_live/index_test.exs -just test-slow --seed 123 -``` - -**Run all tests (fast + slow):** -```bash -just test -# or -mix test - -# With specific files or options -just test-all test/mv_web/ -just test-all --max-failures 5 -``` - -**Note:** All suite commands (`test-fast`, `test-slow`, `test-all`) support additional arguments. The suite semantics (tags) are always preserved - additional arguments are appended to the command. - -### CI/CD - -**Standard CI Pipeline:** -- Runs `mix test --exclude slow` for faster feedback -- Executes on every push to any branch - -**Nightly Pipeline:** -- Runs `mix test --only slow` for comprehensive performance coverage -- Executes daily at 2 AM via cron trigger -- Pipeline name: `nightly-tests` - -**Manual Execution:** - -The nightly pipeline can be manually triggered using: - -**Drone CLI:** -```bash -drone build start / --event custom -``` - -**Drone Web UI:** -- Navigate to the repository in Drone -- Go to the `nightly-tests` pipeline -- Click "Run" or "Restart" and select event type `custom` - -**Example:** -```bash -# Trigger nightly-tests pipeline manually -drone build start local-it/mitgliederverwaltung main --event custom -``` - ---- - -## Best Practices for New Performance Tests - -### When to Tag as `:slow` - -Tag tests as `:slow` when they: - -1. **Explicitly test performance:** Tests that measure execution time or validate performance characteristics -2. **Use large datasets:** Tests that create many records (e.g., 50+ members, 100+ records) -3. **Test scalability:** Tests that verify the application handles larger workloads -4. **Validate query optimization:** Tests that ensure N+1 queries are prevented - -### When NOT to Tag as `:slow` - -Do NOT tag tests as `:slow` if they are: - -1. **Simply slow by accident:** Tests that are slow due to inefficient setup, not intentional performance testing -2. **Slow due to bugs:** Tests that are slow because of actual performance bugs (fix the bug instead) -3. **Integration tests:** Integration tests should be tagged separately if needed (`@tag :integration`) - -### Tagging Guidelines - -**For single tests:** -```elixir -@tag :slow -test "boolean filter performance with 150 members" do - # test implementation -end -``` - -**For describe blocks (all tests in block):** -```elixir -@moduletag :slow -describe "performance" do - test "page loads efficiently" do - # test implementation - end - - test "member count is efficient" do - # test implementation - end -end -``` - ---- - -## Performance Test Structure - -### Recommended Structure - -```elixir -defmodule MvWeb.SomeLiveViewTest do - use MvWeb.ConnCase, async: true - - # Regular tests here (not tagged) - - @moduletag :slow - describe "performance" do - test "loads efficiently without N+1 queries" do - # Create test data - # Measure/validate performance - # Assert correct behavior - end - - test "handles large datasets efficiently" do - # Create large dataset - # Measure performance - # Assert performance requirements - end - end -end -``` - -### Performance Assertions - -Performance tests should include explicit performance assertions: - -```elixir -# Example: Time-based assertion -start_time = System.monotonic_time(:millisecond) -# ... perform operation ... -end_time = System.monotonic_time(:millisecond) -duration = end_time - start_time - -assert duration < 1000, "Operation took #{duration}ms, expected < 1000ms" -``` - -```elixir -# Example: Query count assertion (using Ecto query logging) -# Verify no N+1 queries by checking query count -``` - ---- - -## Monitoring and Maintenance - -### Regular Review - -- **Quarterly Review:** Review slow tests quarterly to ensure they're still relevant -- **Performance Baselines:** Update performance assertions if legitimate performance improvements occur -- **Test Cleanup:** Remove or optimize tests that become redundant - -### Success Metrics - -- ✅ Performance tests catch regressions before production -- ✅ Standard test runs complete in < 3 minutes -- ✅ Nightly builds complete successfully -- ✅ No false positives in performance tests - -### Troubleshooting - -**If a performance test fails:** - -1. **Check if it's a real regression:** Compare with previous runs -2. **Check CI environment:** Ensure CI has adequate resources -3. **Review test data:** Ensure test data setup is correct -4. **Check for flakiness:** Run test multiple times to verify consistency - -**If a performance test is too slow:** - -1. **Review test implementation:** Look for inefficiencies in test setup -2. **Consider reducing dataset size:** If still representative -3. **Split into smaller tests:** If testing multiple concerns - ---- - -## Related Documentation - -- **Test Optimization Summary:** `docs/test-optimization-summary.md` -- **Seeds Test Optimization:** `docs/test-optimization-seeds.md` -- **Testing Standards:** `CODE_GUIDELINES.md` - Section 4 (Testing Standards) -- **CI/CD Configuration:** `.drone.yml` - ---- - -## Changelog - -### 2026-01-28: Initial Setup -- Marked 5 performance test suites with `@tag :slow` or `@moduletag :slow` -- Added `test-fast`, `test-slow`, and `test-all` commands to Justfile -- Updated CI to exclude slow tests from standard runs -- Added nightly CI pipeline for slow tests -- Created this documentation diff --git a/test/mv_web/member_live/index_member_fields_display_test.exs b/test/mv_web/member_live/index_member_fields_display_test.exs index ca6ffb0..14c8374 100644 --- a/test/mv_web/member_live/index_member_fields_display_test.exs +++ b/test/mv_web/member_live/index_member_fields_display_test.exs @@ -56,7 +56,9 @@ defmodule MvWeb.MemberLive.IndexMemberFieldsDisplayTest do }) conn = conn_with_oidc_user(conn) - {:ok, _view, html} = live(conn, "/members") + # Use search query to filter to only the expected member (Alice) + # This significantly improves test performance by avoiding loading all members from other tests + {:ok, _view, html} = live(conn, "/members?query=Alice") assert html =~ "Email" assert html =~ m.email