test: optimize single test and update docs

docs/test-performance-optimization.md

@@ -0,0 +1,543 @@
+# Test Performance Optimization
+
+**Last Updated:** 2026-01-28  
+**Status:** ✅ Active optimization program
+
+---
+
+## Executive Summary
+
+This document provides a comprehensive overview of test performance optimizations, risk assessments, and future opportunities. The test suite execution time has been reduced through systematic analysis and targeted optimizations.
+
+### Current Performance Metrics
+
+| Metric | Value |
+|--------|-------|
+| **Total Execution Time** (without `:slow` tests) | ~614 seconds (~10.2 minutes) |
+| **Total Tests** | 1,368 tests (+ 25 doctests) |
+| **Async Execution** | 317.7 seconds |
+| **Sync Execution** | 296.2 seconds |
+| **Slow Tests Excluded** | 9 tests (tagged with `@tag :slow`) |
+| **Top 20 Slowest Tests** | 81.2 seconds (13.2% of total time) |
+
+### Optimization Impact Summary
+
+| Optimization | Tests Affected | Time Saved | Status |
+|--------------|----------------|------------|--------|
+| Seeds tests reduction | 13 → 4 tests | ~10-16s | ✅ Completed |
+| Performance tests tagging | 9 tests | ~3-4s per run | ✅ Completed |
+| Critical test query filtering | 1 test | ~8-10s | ✅ Completed |
+| **Total Saved** | | **~21-30s** | |
+
+---
+
+## Completed Optimizations
+
+### 1. Seeds Test Suite Optimization
+
+**Date:** 2026-01-28  
+**Status:** ✅ Completed
+
+#### What Changed
+
+- **Reduced test count:** From 13 tests to 4 tests (69% reduction)
+- **Reduced seeds executions:** From 8-10 times to 5 times per test run
+- **Execution time:** From 24-30 seconds to 13-17 seconds
+- **Time saved:** ~10-16 seconds per test run (40-50% faster)
+
+#### Removed Tests (9 tests)
+
+Tests were removed because their functionality is covered by domain-specific test suites:
+
+1. `"at least one member has no membership fee type assigned"` → Covered by `membership_fees/*_test.exs`
+2. `"each membership fee type has at least one member"` → Covered by `membership_fees/*_test.exs`
+3. `"members with fee types have cycles with various statuses"` → Covered by `cycle_generator_test.exs`
+4. `"creates all 5 authorization roles with correct permission sets"` → Covered by `authorization/*_test.exs`
+5. `"all roles have valid permission_set_names"` → Covered by `authorization/permission_sets_test.exs`
+6. `"does not change role of users who already have a role"` → Merged into idempotency test
+7. `"role creation is idempotent"` (detailed) → Merged into general idempotency test
+
+#### Retained Tests (4 tests)
+
+Critical deployment requirements are still covered:
+
+1. ✅ **Smoke Test:** Seeds run successfully and create basic data
+2. ✅ **Idempotency Test:** Seeds can be run multiple times without duplicating data
+3. ✅ **Admin Bootstrap:** Admin user exists with Admin role (critical for initial access)
+4. ✅ **System Role Bootstrap:** Mitglied system role exists (critical for user registration)
+
+#### Risk Assessment
+
+| Removed Test Category | Alternative Coverage | Risk Level |
+|----------------------|---------------------|------------|
+| Member/fee type distribution | `membership_fees/*_test.exs` | ⚠️ Low |
+| Cycle status variations | `cycle_generator_test.exs` | ⚠️ Low |
+| Detailed role configs | `authorization/*_test.exs` | ⚠️ Very Low |
+| Permission set validation | `permission_sets_test.exs` | ⚠️ Very Low |
+
+**Overall Risk:** ⚠️ **Low** - All removed tests have equivalent or better coverage in domain-specific test suites.
+
+---
+
+### 2. Performance Test Suite Separation
+
+**Date:** 2026-01-28  
+**Status:** ✅ Completed
+
+#### What Changed
+
+Performance tests that explicitly validate performance characteristics are now tagged with `@tag :slow` or `@moduletag :slow` and excluded from standard test runs.
+
+#### Identified Performance Tests (9 tests)
+
+1. **Member LiveView - Boolean Filter Performance** (`test/mv_web/member_live/index_test.exs`)
+   - Test: `"boolean filter performance with 150 members"`
+   - Duration: ~3.8 seconds
+   - Creates 150 members to test filter performance
+
+2. **Group LiveView - Index Performance** (`test/mv_web/live/group_live/index_test.exs`)
+   - 2 tests validating efficient page loading and member count calculation
+
+3. **Group LiveView - Show Performance** (`test/mv_web/live/group_live/show_test.exs`)
+   - 2 tests validating efficient member list loading and slug lookup
+
+4. **Member LiveView - Membership Fee Status Performance** (`test/mv_web/member_live/index_membership_fee_status_test.exs`)
+   - 1 test validating efficient cycle loading without N+1 queries
+
+5. **Group Integration - Query Performance** (`test/membership/group_integration_test.exs`)
+   - 1 test validating N+1 query prevention for group-member relationships
+
+#### Execution Commands
+
+**Fast Tests (Default):**
+```bash
+just test-fast
+# or
+mix test --exclude slow
+```
+
+**Performance Tests Only:**
+```bash
+just test-slow
+# or
+mix test --only slow
+```
+
+**All Tests:**
+```bash
+just test
+# or
+mix test
+```
+
+#### CI/CD Integration
+
+- **Standard CI:** Runs `mix test --exclude slow` for faster feedback loops
+- **Nightly Builds:** Separate pipeline (`nightly-tests`) runs daily at 2 AM and executes all performance tests
+- **Manual Execution:** Can be triggered via Drone CLI or Web UI
+
+#### Risk Assessment
+
+**Risk Level:** ✅ **Very Low**
+
+- Performance tests are still executed, just separately
+- Standard test runs are faster, improving developer feedback
+- Nightly builds ensure comprehensive coverage
+- No functionality is lost, only execution timing changed
+
+---
+
+### 3. Critical Test Optimization
+
+**Date:** 2026-01-28  
+**Status:** ✅ Completed
+
+#### Problem Identified
+
+The test `test respects show_in_overview config` was the slowest test in the suite:
+- **Isolated execution:** 4.8 seconds
+- **In full test run:** 14.7 seconds
+- **Difference:** 9.9 seconds (test isolation issue)
+
+#### Root Cause
+
+The test loaded **all members** from the database, not just the 2 members from the test setup. In full test runs, many members from other tests were present in the database, significantly slowing down the query.
+
+#### Solution Implemented
+
+**Query Filtering:** Added search query parameter to filter to only the expected member.
+
+**Code Change:**
+```elixir
+# Before:
+{:ok, _view, html} = live(conn, "/members")
+
+# After:
+{:ok, _view, html} = live(conn, "/members?query=Alice")
+```
+
+#### Results
+
+| Execution | Before | After | Improvement |
+|-----------|--------|-------|-------------|
+| **Isolated** | 4.8s | 1.1s | **-77%** (3.7s saved) |
+| **In Module** | 4.2s | 0.4s | **-90%** (3.8s saved) |
+| **Expected in Full Run** | 14.7s | ~4-6s | **-65% to -73%** (8-10s saved) |
+
+#### Risk Assessment
+
+**Risk Level:** ✅ **Very Low**
+
+- Test functionality unchanged - only loads expected data
+- All assertions still pass
+- Test is now faster and more isolated
+- No impact on test coverage
+
+---
+
+## Current Performance Analysis
+
+### Top 20 Slowest Tests (without `:slow`)
+
+| Rank | Test | File | Time | % of Top 20 |
+|------|------|------|------|-------------|
+| 1 | `test respects show_in_overview config` | `index_member_fields_display_test.exs` | **~4-6s** (optimized) | ~6-7% |
+| 2 | `test Seeds script is idempotent when run multiple times` | `seeds_test.exs` | 5.0s | 6.2% |
+| 3 | `test sorting functionality initially sorts by email ascending` | `user_live/index_test.exs` | 4.5s | 5.5% |
+| 4 | `test Seeds script runs successfully and creates basic data` | `seeds_test.exs` | 4.3s | 5.3% |
+| 5-20 | Various User LiveView and Policy tests | Multiple files | 2.6-4.2s each | 3-5% each |
+
+**Total Top 20:** ~81 seconds (13.2% of total time)
+
+### Performance Hotspots Identified
+
+#### 1. Seeds Tests (~16.2s for 4 tests)
+
+**Status:** ✅ Optimized (reduced from 13 tests)  
+**Remaining Optimization Potential:** 3-5 seconds
+
+**Opportunities:**
+- Settings update could potentially be moved to `setup_all` (if sandbox allows)
+- Seeds execution could be further optimized (less data in test mode)
+- Idempotency test could be optimized (only 1x seeds instead of 2x)
+
+#### 2. User LiveView Tests (~35.5s for 10 tests)
+
+**Status:** ⏳ Identified for optimization  
+**Optimization Potential:** 15-20 seconds
+
+**Files:**
+- `test/mv_web/user_live/index_test.exs` (3 tests, ~10.2s)
+- `test/mv_web/user_live/form_test.exs` (4 tests, ~15.0s)
+- `test/mv_web/user_live/show_test.exs` (3 tests, ~10.3s)
+
+**Patterns:**
+- Many tests create user/member data
+- LiveView mounts are expensive
+- Form submissions with validations are slow
+
+**Recommended Actions:**
+- Move shared fixtures to `setup_all`
+- Reduce test data volume (3-5 users instead of 10+)
+- Optimize setup patterns for recurring patterns
+
+#### 3. Policy Tests (~8.7s for 3 tests)
+
+**Status:** ⏳ Identified for optimization  
+**Optimization Potential:** 5-8 seconds
+
+**Files:**
+- `test/mv/membership/member_policies_test.exs` (2 tests, ~6.1s)
+- `test/mv/accounts/user_policies_test.exs` (1 test, ~2.6s)
+
+**Pattern:**
+- Each test creates new roles/users/members
+- Roles are identical across tests
+
+**Recommended Actions:**
+- Create roles in `setup_all` (shared across tests)
+- Reuse common fixtures
+- Maintain test isolation while optimizing setup
+
+---
+
+## Future Optimization Opportunities
+
+### Priority 1: User LiveView Tests Optimization
+
+**Estimated Savings:** 15-20 seconds
+
+**Actions:**
+1. Move shared fixtures to `setup_all` where possible
+2. Reduce test data volume (use 3-5 users instead of 10+)
+3. Analyze and optimize recurring setup patterns
+4. Consider mocking expensive operations (if appropriate)
+
+**Risk Assessment:** ⚠️ **Low**
+- Requires careful testing to ensure isolation is maintained
+- Should verify that shared fixtures don't cause test interdependencies
+
+### Priority 2: Policy Tests Optimization
+
+**Estimated Savings:** 5-8 seconds
+
+**Actions:**
+1. Create roles in `setup_all` (they're identical across tests)
+2. Reuse common fixtures (users, members)
+3. Maintain test isolation while optimizing setup
+
+**Risk Assessment:** ⚠️ **Low**
+- Roles are read-only in tests, safe to share
+- Need to ensure user/member fixtures don't interfere with each other
+
+### Priority 3: Seeds Tests Further Optimization
+
+**Estimated Savings:** 3-5 seconds
+
+**Actions:**
+1. Investigate if settings update can be moved to `setup_all`
+2. Introduce seeds mode for tests (less data in test mode)
+3. Optimize idempotency test (only 1x seeds instead of 2x)
+
+**Risk Assessment:** ⚠️ **Low to Medium**
+- Sandbox limitations may prevent `setup_all` usage
+- Seeds mode would require careful implementation
+- Idempotency test optimization needs to maintain test validity
+
+### Priority 4: Additional Test Isolation Improvements
+
+**Estimated Savings:** Variable (depends on specific tests)
+
+**Actions:**
+1. Review tests that load all records (similar to the critical test fix)
+2. Add query filters where appropriate
+3. Ensure proper test isolation in async tests
+
+**Risk Assessment:** ⚠️ **Very Low**
+- Similar to the critical test optimization (proven approach)
+- Improves test isolation and reliability
+
+---
+
+## Estimated Total Optimization Potential
+
+| Priority | Optimization | Estimated Savings |
+|----------|-------------|-------------------|
+| 1 | User LiveView Tests | 15-20s |
+| 2 | Policy Tests | 5-8s |
+| 3 | Seeds Tests Further | 3-5s |
+| 4 | Additional Isolation | Variable |
+| **Total Potential** | | **23-33 seconds** |
+
+**Projected Final Time:** From ~614 seconds to **~580-590 seconds** (~9.5-10 minutes)
+
+---
+
+## Risk Assessment Summary
+
+### Overall Risk Level: ⚠️ **Low**
+
+All optimizations maintain test coverage while improving performance:
+
+| Optimization | Risk Level | Mitigation |
+|-------------|------------|------------|
+| Seeds tests reduction | ⚠️ Low | Coverage mapped to domain tests |
+| Performance tests tagging | ✅ Very Low | Tests still executed, just separately |
+| Critical test optimization | ✅ Very Low | Functionality unchanged, better isolation |
+| Future optimizations | ⚠️ Low | Careful implementation with verification |
+
+### Monitoring Plan
+
+#### Success Criteria
+
+- ✅ Seeds tests execute in <20 seconds consistently
+- ✅ No increase in seeds-related deployment failures
+- ✅ No regression in authorization or membership fee bugs
+- ⏳ Top 20 slowest tests: < 60 seconds (currently 81.2s)
+- ⏳ Total execution time (without `:slow`): < 10 minutes (currently 10.2 min)
+
+#### What to Watch For
+
+1. **Production Seeds Failures:**
+   - Monitor deployment logs for seeds errors
+   - If failures increase, consider restoring detailed tests
+
+2. **Authorization Bugs After Seeds Changes:**
+   - If role/permission bugs appear after seeds modifications
+   - May indicate need for more seeds-specific role validation
+
+3. **Test Performance Regression:**
+   - Monitor test execution times in CI
+   - Alert if times increase significantly
+
+4. **Developer Feedback:**
+   - If developers report missing test coverage
+   - Adjust based on real-world experience
+
+---
+
+## Benchmarking and Analysis
+
+### How to Benchmark Tests
+
+**ExUnit Built-in Benchmarking:**
+
+The test suite is configured to show the slowest tests automatically:
+
+```elixir
+# test/test_helper.exs
+ExUnit.start(
+  slowest: 10  # Shows 10 slowest tests at the end of test run
+)
+```
+
+**Run Benchmark Analysis:**
+
+```bash
+# Show slowest tests
+mix test --slowest 20
+
+# Exclude slow tests for faster feedback
+mix test --exclude slow --slowest 20
+
+# Run only slow tests
+mix test --only slow --slowest 10
+
+# Benchmark specific test file
+mix test test/mv_web/member_live/index_member_fields_display_test.exs --slowest 5
+```
+
+### Benchmarking Best Practices
+
+1. **Run benchmarks regularly** (e.g., monthly) to catch performance regressions
+2. **Compare isolated vs. full runs** to identify test isolation issues
+3. **Monitor CI execution times** to track trends over time
+4. **Document significant changes** in test performance
+
+---
+
+## Test Suite Structure
+
+### Test Execution Modes
+
+**Fast Tests (Default):**
+- Excludes performance tests (`@tag :slow`)
+- Used for standard development workflow
+- Command: `mix test --exclude slow` or `just test-fast`
+
+**Performance Tests:**
+- Explicitly tagged performance tests
+- Run separately or in nightly builds
+- Command: `mix test --only slow` or `just test-slow`
+
+**All Tests:**
+- Includes both fast and slow tests
+- Used for comprehensive validation
+- Command: `mix test` or `just test`
+
+### Test Organization
+
+Tests are organized to mirror the `lib/` directory structure:
+
+```
+test/
+├── accounts/              # Accounts domain tests
+├── membership/            # Membership domain tests
+├── membership_fees/      # Membership fees domain tests
+├── mv/                    # Core application tests
+│   ├── accounts/         # User-related tests
+│   ├── membership/       # Member-related tests
+│   └── authorization/    # Authorization tests
+├── mv_web/               # Web layer tests
+│   ├── controllers/     # Controller tests
+│   ├── live/             # LiveView tests
+│   └── components/       # Component tests
+└── support/              # Test helpers
+    ├── conn_case.ex      # Controller test setup
+    └── data_case.ex      # Database test setup
+```
+
+---
+
+## Best Practices for Test Performance
+
+### When Writing New Tests
+
+1. **Use `async: true`** when possible (for parallel execution)
+2. **Filter queries** to only load necessary data
+3. **Share fixtures** in `setup_all` when appropriate
+4. **Tag performance tests** with `@tag :slow` if they use large datasets
+5. **Keep test data minimal** - only create what's needed for the test
+
+### When Optimizing Existing Tests
+
+1. **Measure first** - Use `mix test --slowest` to identify bottlenecks
+2. **Compare isolated vs. full runs** - Identify test isolation issues
+3. **Optimize setup** - Move shared data to `setup_all` where possible
+4. **Filter queries** - Only load data needed for the test
+5. **Verify coverage** - Ensure optimizations don't reduce test coverage
+
+### Performance Test Guidelines
+
+**Tag as `:slow` when:**
+- Explicitly testing performance characteristics
+- Using large datasets (50+ records)
+- Testing scalability or query optimization
+- Validating N+1 query prevention
+
+**Do NOT tag as `:slow` when:**
+- Test is slow due to inefficient setup (fix the setup instead)
+- Test is slow due to bugs (fix the bug instead)
+- It's an integration test (use `@tag :integration` instead)
+
+---
+
+## Changelog
+
+### 2026-01-28: Initial Optimization Phase
+
+**Completed:**
+- ✅ Reduced seeds tests from 13 to 4 tests
+- ✅ Tagged 9 performance tests with `@tag :slow`
+- ✅ Optimized critical test with query filtering
+- ✅ Created slow test suite infrastructure
+- ✅ Updated CI/CD to exclude slow tests from standard runs
+- ✅ Added nightly CI pipeline for slow tests
+
+**Time Saved:** ~21-30 seconds per test run
+
+**Next Steps:**
+- ⏳ Optimize User LiveView tests (Priority 1)
+- ⏳ Optimize Policy tests (Priority 2)
+- ⏳ Further optimize Seeds tests (Priority 3)
+
+---
+
+## References
+
+- **Testing Standards:** `CODE_GUIDELINES.md` - Section 4 (Testing Standards)
+- **CI/CD Configuration:** `.drone.yml`
+- **Test Helper:** `test/test_helper.exs`
+- **Justfile Commands:** `Justfile` (test-fast, test-slow, test-all)
+
+---
+
+## Questions & Answers
+
+**Q: What if seeds create wrong data and break the system?**  
+A: The smoke test will fail if seeds raise errors. Domain tests ensure business logic is correct regardless of seeds content.
+
+**Q: What if we add a new critical bootstrap requirement?**  
+A: Add a new test to the "Critical bootstrap invariants" section in `test/seeds_test.exs`.
+
+**Q: How do we know the removed tests aren't needed?**  
+A: Monitor for 2-3 months. If no seeds-related bugs appear that would have been caught by removed tests, they were redundant.
+
+**Q: Should we restore the tests for important releases?**  
+A: Consider running the full test suite (including slow tests) before major releases. Daily development uses the optimized suite.
+
+**Q: How do I add a new performance test?**  
+A: Tag it with `@tag :slow` or `@moduletag :slow`. See "Performance Test Guidelines" section above.
+
+**Q: Can I run slow tests locally?**  
+A: Yes, use `just test-slow` or `mix test --only slow`. They're excluded from standard runs for faster feedback.