mitgliederverwaltung/docs/test-performance-optimization.md

# Test Performance Optimization

**Last Updated:** 2026-01-28  
**Status:** ✅ Active optimization program

---

## Executive Summary

This document provides a comprehensive overview of test performance optimizations, risk assessments, and future opportunities. The test suite execution time has been reduced through systematic analysis and targeted optimizations.

### Current Performance Metrics

| Metric | Value |
|--------|-------|
| **Total Execution Time** (without `:slow` tests) | ~614 seconds (~10.2 minutes) |
| **Total Tests** | 1,368 tests (+ 25 doctests) |
| **Async Execution** | 317.7 seconds |
| **Sync Execution** | 296.2 seconds |
| **Slow Tests Excluded** | 9 tests (tagged with `@tag :slow`) |
| **Top 20 Slowest Tests** | 81.2 seconds (13.2% of total time) |

### Optimization Impact Summary

| Optimization | Tests Affected | Time Saved | Status |
|--------------|----------------|------------|--------|
| Seeds tests reduction | 13 → 4 tests | ~10-16s | ✅ Completed |
| Performance tests tagging | 9 tests | ~3-4s per run | ✅ Completed |
| Critical test query filtering | 1 test | ~8-10s | ✅ Completed |
| **Total Saved** | | **~21-30s** | |

---

## Completed Optimizations

### 1. Seeds Test Suite Optimization

**Date:** 2026-01-28  
**Status:** ✅ Completed

#### What Changed

- **Reduced test count:** From 13 tests to 4 tests (69% reduction)
- **Reduced seeds executions:** From 8-10 times to 5 times per test run
- **Execution time:** From 24-30 seconds to 13-17 seconds
- **Time saved:** ~10-16 seconds per test run (40-50% faster)

#### Removed Tests (9 tests)

Tests were removed because their functionality is covered by domain-specific test suites:

1. `"at least one member has no membership fee type assigned"` → Covered by `membership_fees/*_test.exs`
2. `"each membership fee type has at least one member"` → Covered by `membership_fees/*_test.exs`
3. `"members with fee types have cycles with various statuses"` → Covered by `cycle_generator_test.exs`
4. `"creates all 5 authorization roles with correct permission sets"` → Covered by `authorization/*_test.exs`
5. `"all roles have valid permission_set_names"` → Covered by `authorization/permission_sets_test.exs`
6. `"does not change role of users who already have a role"` → Merged into idempotency test
7. `"role creation is idempotent"` (detailed) → Merged into general idempotency test

#### Retained Tests (4 tests)

Critical deployment requirements are still covered:

1. ✅ **Smoke Test:** Seeds run successfully and create basic data
2. ✅ **Idempotency Test:** Seeds can be run multiple times without duplicating data
3. ✅ **Admin Bootstrap:** Admin user exists with Admin role (critical for initial access)
4. ✅ **System Role Bootstrap:** Mitglied system role exists (critical for user registration)

#### Risk Assessment

| Removed Test Category | Alternative Coverage | Risk Level |
|----------------------|---------------------|------------|
| Member/fee type distribution | `membership_fees/*_test.exs` | ⚠️ Low |
| Cycle status variations | `cycle_generator_test.exs` | ⚠️ Low |
| Detailed role configs | `authorization/*_test.exs` | ⚠️ Very Low |
| Permission set validation | `permission_sets_test.exs` | ⚠️ Very Low |

**Overall Risk:** ⚠️ **Low** - All removed tests have equivalent or better coverage in domain-specific test suites.

---

### 2. Performance Test Suite Separation

**Date:** 2026-01-28  
**Status:** ✅ Completed

#### What Changed

Performance tests that explicitly validate performance characteristics are now tagged with `@tag :slow` or `@moduletag :slow` and excluded from standard test runs.

#### Identified Performance Tests (9 tests)

1. **Member LiveView - Boolean Filter Performance** (`test/mv_web/member_live/index_test.exs`)
   - Test: `"boolean filter performance with 150 members"`
   - Duration: ~3.8 seconds
   - Creates 150 members to test filter performance

2. **Group LiveView - Index Performance** (`test/mv_web/live/group_live/index_test.exs`)
   - 2 tests validating efficient page loading and member count calculation

3. **Group LiveView - Show Performance** (`test/mv_web/live/group_live/show_test.exs`)
   - 2 tests validating efficient member list loading and slug lookup

4. **Member LiveView - Membership Fee Status Performance** (`test/mv_web/member_live/index_membership_fee_status_test.exs`)
   - 1 test validating efficient cycle loading without N+1 queries

5. **Group Integration - Query Performance** (`test/membership/group_integration_test.exs`)
   - 1 test validating N+1 query prevention for group-member relationships

#### Execution Commands

**Fast Tests (Default):**
```bash
just test-fast
# or
mix test --exclude slow
```

**Performance Tests Only:**
```bash
just test-slow
# or
mix test --only slow
```

**All Tests:**
```bash
just test
# or
mix test
```

#### CI/CD Integration

- **Standard CI:** Runs `mix test --exclude slow` for faster feedback loops
- **Nightly Builds:** Separate pipeline (`nightly-tests`) runs daily at 2 AM and executes all performance tests
- **Manual Execution:** Can be triggered via Drone CLI or Web UI

#### Risk Assessment

**Risk Level:** ✅ **Very Low**

- Performance tests are still executed, just separately
- Standard test runs are faster, improving developer feedback
- Nightly builds ensure comprehensive coverage
- No functionality is lost, only execution timing changed

---

### 3. Critical Test Optimization

**Date:** 2026-01-28  
**Status:** ✅ Completed

#### Problem Identified

The test `test respects show_in_overview config` was the slowest test in the suite:
- **Isolated execution:** 4.8 seconds
- **In full test run:** 14.7 seconds
- **Difference:** 9.9 seconds (test isolation issue)

#### Root Cause

The test loaded **all members** from the database, not just the 2 members from the test setup. In full test runs, many members from other tests were present in the database, significantly slowing down the query.

#### Solution Implemented

**Query Filtering:** Added search query parameter to filter to only the expected member.

**Code Change:**
```elixir
# Before:
{:ok, _view, html} = live(conn, "/members")

# After:
{:ok, _view, html} = live(conn, "/members?query=Alice")
```

#### Results

| Execution | Before | After | Improvement |
|-----------|--------|-------|-------------|
| **Isolated** | 4.8s | 1.1s | **-77%** (3.7s saved) |
| **In Module** | 4.2s | 0.4s | **-90%** (3.8s saved) |
| **Expected in Full Run** | 14.7s | ~4-6s | **-65% to -73%** (8-10s saved) |

#### Risk Assessment

**Risk Level:** ✅ **Very Low**

- Test functionality unchanged - only loads expected data
- All assertions still pass
- Test is now faster and more isolated
- No impact on test coverage

---

## Current Performance Analysis

### Top 20 Slowest Tests (without `:slow`)

| Rank | Test | File | Time | % of Top 20 |
|------|------|------|------|-------------|
| 1 | `test respects show_in_overview config` | `index_member_fields_display_test.exs` | **~4-6s** (optimized) | ~6-7% |
| 2 | `test Seeds script is idempotent when run multiple times` | `seeds_test.exs` | 5.0s | 6.2% |
| 3 | `test sorting functionality initially sorts by email ascending` | `user_live/index_test.exs` | 4.5s | 5.5% |
| 4 | `test Seeds script runs successfully and creates basic data` | `seeds_test.exs` | 4.3s | 5.3% |
| 5-20 | Various User LiveView and Policy tests | Multiple files | 2.6-4.2s each | 3-5% each |

**Total Top 20:** ~81 seconds (13.2% of total time)

### Performance Hotspots Identified

#### 1. Seeds Tests (~16.2s for 4 tests)

**Status:** ✅ Optimized (reduced from 13 tests)  
**Remaining Optimization Potential:** 3-5 seconds

**Opportunities:**
- Settings update could potentially be moved to `setup_all` (if sandbox allows)
- Seeds execution could be further optimized (less data in test mode)
- Idempotency test could be optimized (only 1x seeds instead of 2x)

#### 2. User LiveView Tests (~35.5s for 10 tests)

**Status:** ⏳ Identified for optimization  
**Optimization Potential:** 15-20 seconds

**Files:**
- `test/mv_web/user_live/index_test.exs` (3 tests, ~10.2s)
- `test/mv_web/user_live/form_test.exs` (4 tests, ~15.0s)
- `test/mv_web/user_live/show_test.exs` (3 tests, ~10.3s)

**Patterns:**
- Many tests create user/member data
- LiveView mounts are expensive
- Form submissions with validations are slow

**Recommended Actions:**
- Move shared fixtures to `setup_all`
- Reduce test data volume (3-5 users instead of 10+)
- Optimize setup patterns for recurring patterns

#### 3. Policy Tests (~8.7s for 3 tests)

**Status:** ⏳ Identified for optimization  
**Optimization Potential:** 5-8 seconds

**Files:**
- `test/mv/membership/member_policies_test.exs` (2 tests, ~6.1s)
- `test/mv/accounts/user_policies_test.exs` (1 test, ~2.6s)

**Pattern:**
- Each test creates new roles/users/members
- Roles are identical across tests

**Recommended Actions:**
- Create roles in `setup_all` (shared across tests)
- Reuse common fixtures
- Maintain test isolation while optimizing setup

---

## Future Optimization Opportunities

### Priority 1: User LiveView Tests Optimization

**Estimated Savings:** 15-20 seconds

**Actions:**
1. Move shared fixtures to `setup_all` where possible
2. Reduce test data volume (use 3-5 users instead of 10+)
3. Analyze and optimize recurring setup patterns
4. Consider mocking expensive operations (if appropriate)

**Risk Assessment:** ⚠️ **Low**
- Requires careful testing to ensure isolation is maintained
- Should verify that shared fixtures don't cause test interdependencies

### Priority 2: Policy Tests Optimization

**Estimated Savings:** 5-8 seconds

**Actions:**
1. Create roles in `setup_all` (they're identical across tests)
2. Reuse common fixtures (users, members)
3. Maintain test isolation while optimizing setup

**Risk Assessment:** ⚠️ **Low**
- Roles are read-only in tests, safe to share
- Need to ensure user/member fixtures don't interfere with each other

### Priority 3: Seeds Tests Further Optimization

**Estimated Savings:** 3-5 seconds

**Actions:**
1. Investigate if settings update can be moved to `setup_all`
2. Introduce seeds mode for tests (less data in test mode)
3. Optimize idempotency test (only 1x seeds instead of 2x)

**Risk Assessment:** ⚠️ **Low to Medium**
- Sandbox limitations may prevent `setup_all` usage
- Seeds mode would require careful implementation
- Idempotency test optimization needs to maintain test validity

### Priority 4: Additional Test Isolation Improvements

**Estimated Savings:** Variable (depends on specific tests)

**Actions:**
1. Review tests that load all records (similar to the critical test fix)
2. Add query filters where appropriate
3. Ensure proper test isolation in async tests

**Risk Assessment:** ⚠️ **Very Low**
- Similar to the critical test optimization (proven approach)
- Improves test isolation and reliability

---

## Estimated Total Optimization Potential

| Priority | Optimization | Estimated Savings |
|----------|-------------|-------------------|
| 1 | User LiveView Tests | 15-20s |
| 2 | Policy Tests | 5-8s |
| 3 | Seeds Tests Further | 3-5s |
| 4 | Additional Isolation | Variable |
| **Total Potential** | | **23-33 seconds** |

**Projected Final Time:** From ~614 seconds to **~580-590 seconds** (~9.5-10 minutes)

---

## Risk Assessment Summary

### Overall Risk Level: ⚠️ **Low**

All optimizations maintain test coverage while improving performance:

| Optimization | Risk Level | Mitigation |
|-------------|------------|------------|
| Seeds tests reduction | ⚠️ Low | Coverage mapped to domain tests |
| Performance tests tagging | ✅ Very Low | Tests still executed, just separately |
| Critical test optimization | ✅ Very Low | Functionality unchanged, better isolation |
| Future optimizations | ⚠️ Low | Careful implementation with verification |

### Monitoring Plan

#### Success Criteria

- ✅ Seeds tests execute in <20 seconds consistently
- ✅ No increase in seeds-related deployment failures
- ✅ No regression in authorization or membership fee bugs
- ⏳ Top 20 slowest tests: < 60 seconds (currently 81.2s)
- ⏳ Total execution time (without `:slow`): < 10 minutes (currently 10.2 min)

#### What to Watch For

1. **Production Seeds Failures:**
   - Monitor deployment logs for seeds errors
   - If failures increase, consider restoring detailed tests

2. **Authorization Bugs After Seeds Changes:**
   - If role/permission bugs appear after seeds modifications
   - May indicate need for more seeds-specific role validation

3. **Test Performance Regression:**
   - Monitor test execution times in CI
   - Alert if times increase significantly

4. **Developer Feedback:**
   - If developers report missing test coverage
   - Adjust based on real-world experience

---

## Benchmarking and Analysis

### How to Benchmark Tests

**ExUnit Built-in Benchmarking:**

The test suite is configured to show the slowest tests automatically:

```elixir
# test/test_helper.exs
ExUnit.start(
  slowest: 10  # Shows 10 slowest tests at the end of test run
)
```

**Run Benchmark Analysis:**

```bash
# Show slowest tests
mix test --slowest 20

# Exclude slow tests for faster feedback
mix test --exclude slow --slowest 20

# Run only slow tests
mix test --only slow --slowest 10

# Benchmark specific test file
mix test test/mv_web/member_live/index_member_fields_display_test.exs --slowest 5
```

### Benchmarking Best Practices

1. **Run benchmarks regularly** (e.g., monthly) to catch performance regressions
2. **Compare isolated vs. full runs** to identify test isolation issues
3. **Monitor CI execution times** to track trends over time
4. **Document significant changes** in test performance

---

## Test Suite Structure

### Test Execution Modes

**Fast Tests (Default):**
- Excludes performance tests (`@tag :slow`)
- Used for standard development workflow
- Command: `mix test --exclude slow` or `just test-fast`

**Performance Tests:**
- Explicitly tagged performance tests
- Run separately or in nightly builds
- Command: `mix test --only slow` or `just test-slow`

**All Tests:**
- Includes both fast and slow tests
- Used for comprehensive validation
- Command: `mix test` or `just test`

### Test Organization

Tests are organized to mirror the `lib/` directory structure:

```
test/
├── accounts/              # Accounts domain tests
├── membership/            # Membership domain tests
├── membership_fees/      # Membership fees domain tests
├── mv/                    # Core application tests
│   ├── accounts/         # User-related tests
│   ├── membership/       # Member-related tests
│   └── authorization/    # Authorization tests
├── mv_web/               # Web layer tests
│   ├── controllers/     # Controller tests
│   ├── live/             # LiveView tests
│   └── components/       # Component tests
└── support/              # Test helpers
    ├── conn_case.ex      # Controller test setup
    └── data_case.ex      # Database test setup
```

---

## Best Practices for Test Performance

### When Writing New Tests

1. **Use `async: true`** when possible (for parallel execution)
2. **Filter queries** to only load necessary data
3. **Share fixtures** in `setup_all` when appropriate
4. **Tag performance tests** with `@tag :slow` if they use large datasets
5. **Keep test data minimal** - only create what's needed for the test

### When Optimizing Existing Tests

1. **Measure first** - Use `mix test --slowest` to identify bottlenecks
2. **Compare isolated vs. full runs** - Identify test isolation issues
3. **Optimize setup** - Move shared data to `setup_all` where possible
4. **Filter queries** - Only load data needed for the test
5. **Verify coverage** - Ensure optimizations don't reduce test coverage

### Performance Test Guidelines

**Tag as `:slow` when:**
- Explicitly testing performance characteristics
- Using large datasets (50+ records)
- Testing scalability or query optimization
- Validating N+1 query prevention

**Do NOT tag as `:slow` when:**
- Test is slow due to inefficient setup (fix the setup instead)
- Test is slow due to bugs (fix the bug instead)
- It's an integration test (use `@tag :integration` instead)

---

## Changelog

### 2026-01-28: Initial Optimization Phase

**Completed:**
- ✅ Reduced seeds tests from 13 to 4 tests
- ✅ Tagged 9 performance tests with `@tag :slow`
- ✅ Optimized critical test with query filtering
- ✅ Created slow test suite infrastructure
- ✅ Updated CI/CD to exclude slow tests from standard runs
- ✅ Added nightly CI pipeline for slow tests

**Time Saved:** ~21-30 seconds per test run

**Next Steps:**
- ⏳ Optimize User LiveView tests (Priority 1)
- ⏳ Optimize Policy tests (Priority 2)
- ⏳ Further optimize Seeds tests (Priority 3)

---

## References

- **Testing Standards:** `CODE_GUIDELINES.md` - Section 4 (Testing Standards)
- **CI/CD Configuration:** `.drone.yml`
- **Test Helper:** `test/test_helper.exs`
- **Justfile Commands:** `Justfile` (test-fast, test-slow, test-all)

---

## Questions & Answers

**Q: What if seeds create wrong data and break the system?**  
A: The smoke test will fail if seeds raise errors. Domain tests ensure business logic is correct regardless of seeds content.

**Q: What if we add a new critical bootstrap requirement?**  
A: Add a new test to the "Critical bootstrap invariants" section in `test/seeds_test.exs`.

**Q: How do we know the removed tests aren't needed?**  
A: Monitor for 2-3 months. If no seeds-related bugs appear that would have been caught by removed tests, they were redundant.

**Q: Should we restore the tests for important releases?**  
A: Consider running the full test suite (including slow tests) before major releases. Daily development uses the optimized suite.

**Q: How do I add a new performance test?**  
A: Tag it with `@tag :slow` or `@moduletag :slow`. See "Performance Test Guidelines" section above.

**Q: Can I run slow tests locally?**  
A: Yes, use `just test-slow` or `mix test --only slow`. They're excluded from standard runs for faster feedback.