mitgliederverwaltung/docs/test-performance-optimization.md

# Test Performance Optimization

**Last Updated:** 2026-01-28  
**Status:** ✅ Active optimization program

---

## Executive Summary

This document provides a comprehensive overview of test performance optimizations, risk assessments, and future opportunities. The test suite execution time has been reduced through systematic analysis and targeted optimizations.

### Current Performance Metrics

| Metric | Value |
|--------|-------|
| **Total Execution Time** (without `:slow` tests) | ~368 seconds (~6.1 minutes) |
| **Total Tests** | 1,336 tests (+ 25 doctests) |
| **Async Execution** | 163.5 seconds |
| **Sync Execution** | 281.5 seconds |
| **Slow Tests Excluded** | 25 tests (tagged with `@tag :slow`) |
| **Top 50 Slowest Tests** | 121.9 seconds (27.4% of total time) |

### Optimization Impact Summary

| Optimization | Tests Affected | Time Saved | Status |
|--------------|----------------|------------|--------|
| Seeds tests reduction | 13 → 4 tests | ~10-16s | ✅ Completed |
| Performance tests tagging | 9 tests | ~3-4s per run | ✅ Completed |
| Critical test query filtering | 1 test | ~8-10s | ✅ Completed |
| Full test suite via promotion | 25 tests | ~77s per run | ✅ Completed |
| **Total Saved** | | **~98-107s** | |

---

## Completed Optimizations

### 1. Seeds Test Suite Optimization

**Date:** 2026-01-28  
**Status:** ✅ Completed

#### What Changed

- **Reduced test count:** From 13 tests to 4 tests (69% reduction)
- **Reduced seeds executions:** From 8-10 times to 5 times per test run
- **Execution time:** From 24-30 seconds to 13-17 seconds
- **Time saved:** ~10-16 seconds per test run (40-50% faster)

#### Removed Tests (9 tests)

Tests were removed because their functionality is covered by domain-specific test suites:

1. `"at least one member has no membership fee type assigned"` → Covered by `membership_fees/*_test.exs`
2. `"each membership fee type has at least one member"` → Covered by `membership_fees/*_test.exs`
3. `"members with fee types have cycles with various statuses"` → Covered by `cycle_generator_test.exs`
4. `"creates all 5 authorization roles with correct permission sets"` → Covered by `authorization/*_test.exs`
5. `"all roles have valid permission_set_names"` → Covered by `authorization/permission_sets_test.exs`
6. `"does not change role of users who already have a role"` → Merged into idempotency test
7. `"role creation is idempotent"` (detailed) → Merged into general idempotency test

#### Retained Tests (4 tests)

Critical deployment requirements are still covered:

1. ✅ **Smoke Test:** Seeds run successfully and create basic data
2. ✅ **Idempotency Test:** Seeds can be run multiple times without duplicating data
3. ✅ **Admin Bootstrap:** Admin user exists with Admin role (critical for initial access)
4. ✅ **System Role Bootstrap:** Mitglied system role exists (critical for user registration)

#### Risk Assessment

| Removed Test Category | Alternative Coverage | Risk Level |
|----------------------|---------------------|------------|
| Member/fee type distribution | `membership_fees/*_test.exs` | ⚠️ Low |
| Cycle status variations | `cycle_generator_test.exs` | ⚠️ Low |
| Detailed role configs | `authorization/*_test.exs` | ⚠️ Very Low |
| Permission set validation | `permission_sets_test.exs` | ⚠️ Very Low |

**Overall Risk:** ⚠️ **Low** - All removed tests have equivalent or better coverage in domain-specific test suites.

---

### 2. Full Test Suite via Promotion (`@tag :slow`)

**Date:** 2026-01-28  
**Status:** ✅ Completed

#### What Changed

Tests with **low risk** and **execution time >1 second** are now tagged with `@tag :slow` and excluded from standard test runs. These tests are important but not critical for every commit and are run via promotion before merging to `main`.

#### Tagging Criteria

**Tagged as `@tag :slow` when:**
- ✅ Test execution time >1 second
- ✅ Low risk (not critical for catching regressions in core business logic)
- ✅ UI/Display tests (formatting, rendering)
- ✅ Workflow detail tests (not core functionality)
- ✅ Edge cases with large datasets

**NOT tagged when:**
- ❌ Core CRUD operations (Member/User Create/Update/Destroy)
- ❌ Basic Authentication/Authorization
- ❌ Critical Bootstrap (Admin user, system roles)
- ❌ Email Synchronization
- ❌ Representative tests per Permission Set + Action

#### Identified Tests for Full Test Suite (25 tests)

**1. Seeds Tests (2 tests) - 18.1s**
- `"runs successfully and creates basic data"` (9.0s)
- `"is idempotent when run multiple times"` (9.1s)
- **Note:** Critical bootstrap tests remain in fast suite

**2. UserLive.ShowTest (3 tests) - 10.8s**
- `"mounts successfully with valid user ID"` (4.2s)
- `"displays linked member when present"` (2.4s)
- `"redirects to user list when viewing system actor user"` (4.2s)

**3. UserLive.IndexTest (5 tests) - 25.0s**
- `"displays users in a table"` (1.0s)
- `"initially sorts by email ascending"` (2.2s)
- `"can sort email descending by clicking sort button"` (3.4s)
- `"select all automatically checks when all individual users are selected"` (2.0s)
- `"displays linked member name in user list"` (1.9s)

**4. MemberLive.IndexCustomFieldsDisplayTest (3 tests) - 4.9s**
- `"displays custom field with show_in_overview: true"` (1.6s)
- `"formats date custom field values correctly"` (1.5s)
- `"formats email custom field values correctly"` (1.8s)

**5. MemberLive.IndexCustomFieldsEdgeCasesTest (3 tests) - 3.6s**
- `"displays custom field column even when no members have values"` (1.1s)
- `"displays very long custom field values correctly"` (1.4s)
- `"handles multiple custom fields with show_in_overview correctly"` (1.2s)

**6. RoleLive Tests (7 tests) - 7.7s**
- `role_live_test.exs`: `"mounts successfully"` (1.5s), `"deletes non-system role"` (2.1s)
- `role_live/show_test.exs`: 5 tests >1s (mount, display, navigation)

**7. MemberAvailableForLinkingTest (1 test) - 1.5s**
- `"limits results to 10 members even when more exist"` (1.5s)

**8. Performance Tests (1 test) - 3.8s**
- `"boolean filter performance with 150 members"` (3.8s)

**Total:** 25 tests, ~77 seconds saved

#### Execution Commands

**Fast Tests (Default):**
```bash
just test-fast
# or
mix test --exclude slow
```

**Slow Tests Only:**
```bash
just test-slow
# or
mix test --only slow
```

**All Tests:**
```bash
just test
# or
mix test
```

#### CI/CD Integration

- **Standard CI (`check-fast`):** Runs `mix test --exclude slow --exclude ui` for faster feedback loops (~6 minutes)
- **Full Test Suite (`check-full`):** Triggered via promotion before merge, executes `mix test` (all tests, including slow and UI) for comprehensive coverage (~7.4 minutes)
- **Pre-Merge:** Full test suite (`mix test`) runs via promotion before merging to main
- **Manual Execution:** Promote build to `production` in Drone CI to trigger full test suite

#### Risk Assessment

**Risk Level:** ✅ **Very Low**

- All tagged tests have **low risk** - they don't catch critical regressions
- Core functionality remains tested (CRUD, Auth, Bootstrap)
- Standard test runs are faster (~6 minutes vs ~7.4 minutes)
- Full test suite runs via promotion before merge ensures comprehensive coverage
- No functionality is lost, only execution timing changed

**Critical Tests Remain in Fast Suite:**
- Core CRUD operations (Member/User Create/Update/Destroy)
- Basic Authentication/Authorization
- Critical Bootstrap (Admin user, system roles)
- Email Synchronization
- Representative Policy tests (one per Permission Set + Action)

---

### 3. Critical Test Optimization

**Date:** 2026-01-28  
**Status:** ✅ Completed

#### Problem Identified

The test `test respects show_in_overview config` was the slowest test in the suite:
- **Isolated execution:** 4.8 seconds
- **In full test run:** 14.7 seconds
- **Difference:** 9.9 seconds (test isolation issue)

#### Root Cause

The test loaded **all members** from the database, not just the 2 members from the test setup. In full test runs, many members from other tests were present in the database, significantly slowing down the query.

#### Solution Implemented

**Query Filtering:** Added search query parameter to filter to only the expected member.

**Code Change:**
```elixir
# Before:
{:ok, _view, html} = live(conn, "/members")

# After:
{:ok, _view, html} = live(conn, "/members?query=Alice")
```

#### Results

| Execution | Before | After | Improvement |
|-----------|--------|-------|-------------|
| **Isolated** | 4.8s | 1.1s | **-77%** (3.7s saved) |
| **In Module** | 4.2s | 0.4s | **-90%** (3.8s saved) |
| **Expected in Full Run** | 14.7s | ~4-6s | **-65% to -73%** (8-10s saved) |

#### Risk Assessment

**Risk Level:** ✅ **Very Low**

- Test functionality unchanged - only loads expected data
- All assertions still pass
- Test is now faster and more isolated
- No impact on test coverage

---

### 3. Full Test Suite Analysis and Categorization

**Date:** 2026-01-28  
**Status:** ✅ Completed

#### Analysis Methodology

A comprehensive analysis was performed to identify tests suitable for the full test suite (via promotion) based on:
- **Execution time:** Tests taking >1 second
- **Risk assessment:** Tests that don't catch critical regressions
- **Test category:** UI/Display, workflow details, edge cases

#### Test Categorization

**🔴 CRITICAL - Must Stay in Fast Suite:**
- Core Business Logic (Member/User CRUD)
- Authentication & Authorization Basics
- Critical Bootstrap (Admin user, system roles)
- Email Synchronization
- Representative Policy Tests (one per Permission Set + Action)

**🟡 LOW RISK - Moved to Full Test Suite (via Promotion):**
- Seeds Tests (non-critical: smoke test, idempotency)
- LiveView Display/Formatting Tests
- UserLive.ShowTest (core functionality covered by Index/Form)
- UserLive.IndexTest UI Features (sorting, checkboxes, navigation)
- RoleLive Tests (role management, not core authorization)
- MemberLive Custom Fields Display Tests
- Edge Cases with Large Datasets

#### Risk Assessment Summary

| Category | Tests | Time Saved | Risk Level | Rationale |
|----------|-------|------------|------------|-----------|
| Seeds (non-critical) | 2 | 18.1s | ⚠️ Low | Critical bootstrap tests remain |
| UserLive.ShowTest | 3 | 10.8s | ⚠️ Low | Core CRUD covered by Index/Form |
| UserLive.IndexTest (UI) | 5 | 25.0s | ⚠️ Low | UI features, not core functionality |
| Custom Fields Display | 6 | 8.5s | ⚠️ Low | Formatting tests, visible in code review |
| RoleLive Tests | 7 | 7.7s | ⚠️ Low | Role management, not authorization |
| Edge Cases | 1 | 1.5s | ⚠️ Low | Edge case, not critical path |
| Performance Tests | 1 | 3.8s | ✅ Very Low | Explicit performance validation |
| **Total** | **25** | **~77s** | **⚠️ Low** | |

**Overall Risk:** ⚠️ **Low** - All moved tests have low risk and don't catch critical regressions. Core functionality remains fully tested.

#### Tests Excluded from Full Test Suite

The following tests were **NOT** moved to full test suite (via promotion) despite being slow:

- **Policy Tests:** Medium risk - kept in fast suite (representative tests remain)
- **UserLive.FormTest:** Medium risk - core CRUD functionality
- **Tests <1s:** Don't meet execution time threshold
- **Critical Bootstrap Tests:** High risk - deployment critical

---

## Current Performance Analysis

### Top 20 Slowest Tests (without `:slow`)

After implementing the full test suite via promotion, the remaining slowest tests are:

| Rank | Test | File | Time | Category |
|------|------|------|------|----------|
| 1 | `test Critical bootstrap invariants Mitglied system role exists` | `seeds_test.exs` | 6.7s | Critical Bootstrap |
| 2 | `test Critical bootstrap invariants Admin user has Admin role` | `seeds_test.exs` | 5.0s | Critical Bootstrap |
| 3 | `test normal_user permission set can read own user record` | `user_policies_test.exs` | 2.6s | Policy Test |
| 4 | `test normal_user permission set can create member` | `member_policies_test.exs` | 2.5s | Policy Test |
| 5-20 | Various Policy and LiveView tests | Multiple files | 1.5-2.4s each | Policy/LiveView |

**Total Top 20:** ~44 seconds (12% of total time without `:slow`)

**Note:** Many previously slow tests (UserLive.IndexTest, UserLive.ShowTest, Display/Formatting tests) are now tagged with `@tag :slow` and excluded from standard runs.

### Performance Hotspots Identified

#### 1. Seeds Tests (~16.2s for 4 tests)

**Status:** ✅ Optimized (reduced from 13 tests)  
**Remaining Optimization Potential:** 3-5 seconds

**Opportunities:**
- Settings update could potentially be moved to `setup_all` (if sandbox allows)
- Seeds execution could be further optimized (less data in test mode)
- Idempotency test could be optimized (only 1x seeds instead of 2x)

#### 2. User LiveView Tests (~35.5s for 10 tests)

**Status:** ⏳ Identified for optimization  
**Optimization Potential:** 15-20 seconds

**Files:**
- `test/mv_web/user_live/index_test.exs` (3 tests, ~10.2s)
- `test/mv_web/user_live/form_test.exs` (4 tests, ~15.0s)
- `test/mv_web/user_live/show_test.exs` (3 tests, ~10.3s)

**Patterns:**
- Many tests create user/member data
- LiveView mounts are expensive
- Form submissions with validations are slow

**Recommended Actions:**
- Move shared fixtures to `setup_all`
- Reduce test data volume (3-5 users instead of 10+)
- Optimize setup patterns for recurring patterns

#### 3. Policy Tests (~8.7s for 3 tests)

**Status:** ⏳ Identified for optimization  
**Optimization Potential:** 5-8 seconds

**Files:**
- `test/mv/membership/member_policies_test.exs` (2 tests, ~6.1s)
- `test/mv/accounts/user_policies_test.exs` (1 test, ~2.6s)

**Pattern:**
- Each test creates new roles/users/members
- Roles are identical across tests

**Recommended Actions:**
- Create roles in `setup_all` (shared across tests)
- Reuse common fixtures
- Maintain test isolation while optimizing setup

---

## Future Optimization Opportunities

### Priority 1: User LiveView Tests Optimization

**Estimated Savings:** 14-22 seconds  
**Status:** 📋 Analysis Complete - Ready for Implementation

#### Analysis Summary

Analysis of User LiveView tests identified significant optimization opportunities:
- **Framework functionality over-testing:** ~30 tests test Phoenix/Ash/Gettext core features
- **Redundant test data creation:** Each test creates users/members independently
- **Missing shared fixtures:** No `setup_all` usage for common data

#### Current Performance

**Top 20 Slowest Tests (User LiveView):**
- `index_test.exs`: ~10.2s for 3 tests in Top 20
- `form_test.exs`: ~15.0s for 4 tests in Top 20
- `show_test.exs`: ~10.3s for 3 tests in Top 20
- **Total:** ~35.5 seconds for User LiveView tests

#### Optimization Opportunities

**1. Remove Framework Functionality Tests (~30 tests, 8-12s saved)**
- Remove translation tests (Gettext framework)
- Remove navigation tests (Phoenix LiveView framework)
- Remove validation tests (Ash framework)
- Remove basic HTML rendering tests (consolidate into smoke test)
- Remove password storage tests (AshAuthentication framework)

**2. Implement Shared Fixtures (3-5s saved)**
- Use `setup_all` for common test data in `index_test.exs` and `show_test.exs`
- Share users for sorting/checkbox tests
- Share common users/members across tests
- **Note:** `form_test.exs` uses `async: false`, preventing `setup_all` usage

**3. Consolidate Redundant Tests (~10 tests → 3-4 tests, 2-3s saved)**
- Merge basic display tests into smoke test
- Merge navigation tests into integration test
- Reduce sorting tests to 1 integration test

**4. Optimize Test Data Volume (1-2s saved)**
- Use minimum required data (2 users for sorting, 2 for checkboxes)
- Share data across tests via `setup_all`

#### Tests to Keep (Business Logic)

**Index Tests:**
- `initially sorts by email ascending` - Tests default sort
- `can sort email descending by clicking sort button` - Tests sort functionality
- `select all automatically checks when all individual users are selected` - Business logic
- `does not show system actor user in list` - Business rule
- `displays linked member name in user list` - Business logic
- Edge case tests

**Form Tests:**
- `creates user without password` - Business logic
- `creates user with password when enabled` - Business logic
- `admin sets new password for user` - Business logic
- `selecting member and saving links member to user` - Business logic
- Member linking/unlinking workflow tests

**Show Tests:**
- `displays password authentication status` - Business logic
- `displays linked member when present` - Business logic
- `redirects to user list when viewing system actor user` - Business rule

#### Implementation Plan

**Phase 1: Remove Framework Tests (1-2 hours, ⚠️ Very Low Risk)**
- Remove translation, navigation, validation, and basic HTML rendering tests
- Consolidate remaining display tests into smoke test

**Phase 2: Implement Shared Fixtures (2-3 hours, ⚠️ Low Risk)**
- Add `setup_all` to `index_test.exs` and `show_test.exs`
- Update tests to use shared fixtures
- Verify test isolation maintained

**Phase 3: Consolidate Tests (1-2 hours, ⚠️ Very Low Risk)**
- Merge basic display tests into smoke test
- Merge navigation tests into integration test
- Reduce sorting tests to 1 integration test

**Risk Assessment:** ⚠️ **Low**
- Framework functionality is tested by framework maintainers
- Business logic tests remain intact
- Shared fixtures maintain test isolation
- Consolidation preserves coverage

### Priority 2: Policy Tests Optimization

**Estimated Savings:** 5.5-9 seconds  
**Status:** 📋 Analysis Complete - Ready for Decision

#### Analysis Summary

Analysis of policy tests identified significant optimization opportunities:
- **Redundant fixture creation:** Roles and users created repeatedly across tests
- **Framework functionality over-testing:** Many tests verify Ash policy framework behavior
- **Test duplication:** Similar tests across different permission sets

#### Current Performance

**Policy Test Files Performance:**
- `member_policies_test.exs`: 24 tests, ~66s (top 20)
- `user_policies_test.exs`: 30 tests, ~66s (top 20)
- `custom_field_value_policies_test.exs`: 20 tests, ~66s (top 20)
- **Total:** 74 tests, ~152s total

**Top 20 Slowest Policy Tests:** ~66 seconds

#### Framework vs. Business Logic Analysis

**Framework Functionality (Should NOT Test):**
- Policy evaluation (how Ash evaluates policies)
- Permission lookup (how Ash looks up permissions)
- Scope filtering (how Ash applies scope filters)
- Auto-filter behavior (how Ash auto-filters queries)
- Forbidden vs NotFound (how Ash returns errors)

**Business Logic (Should Test):**
- Permission set definitions (what permissions each role has)
- Scope definitions (what scopes each permission set uses)
- Special cases (custom business rules)
- Permission set behavior (how our permission sets differ)

#### Optimization Opportunities

**1. Remove Framework Functionality Tests (~22-34 tests, 3-4s saved)**
- Remove "cannot" tests that verify error types (Forbidden, NotFound)
- Remove tests that verify auto-filter behavior (framework)
- Remove tests that verify permission evaluation (framework)
- **Risk:** ⚠️ Very Low - Framework functionality is tested by Ash maintainers

**2. Consolidate Redundant Tests (~6-8 tests → 2-3 tests, 1-2s saved)**
- Merge similar tests across permission sets
- Create integration tests that cover multiple permission sets
- **Risk:** ⚠️ Low - Same coverage, fewer tests

**3. Share Admin User Across Describe Blocks (1-2s saved)**
- Create admin user once in module-level `setup`
- Reuse admin user in helper functions
- **Note:** `async: false` prevents `setup_all`, but module-level `setup` works
- **Risk:** ⚠️ Low - Admin user is read-only in tests, safe to share

**4. Reduce Test Data Volume (0.5-1s saved)**
- Use minimum required data
- Share fixtures where possible
- **Risk:** ⚠️ Very Low - Still tests same functionality

#### Test Classification Summary

**Tests to Remove (Framework):**
- `member_policies_test.exs`: ~10 tests (cannot create/destroy/update, auto-filter tests)
- `user_policies_test.exs`: ~16 tests (cannot read/update/create/destroy, auto-filter tests)
- `custom_field_value_policies_test.exs`: ~8 tests (similar "cannot" tests)

**Tests to Consolidate (Redundant):**
- `user_policies_test.exs`: 6 tests → 2 tests (can read/update own user record)

**Tests to Keep (Business Logic):**
- All "can" tests that verify permission set behavior
- Special case tests (e.g., "user can always READ linked member")
- AshAuthentication bypass tests (our integration)

#### Implementation Plan

**Phase 1: Remove Framework Tests (1-2 hours, ⚠️ Very Low Risk)**
- Identify all "cannot" tests that verify error types
- Remove tests that verify Ash auto-filter behavior
- Remove tests that verify permission evaluation (framework)

**Phase 2: Consolidate Redundant Tests (1-2 hours, ⚠️ Low Risk)**
- Identify similar tests across permission sets
- Create integration tests that cover multiple permission sets
- Remove redundant individual tests

**Phase 3: Share Admin User (1-2 hours, ⚠️ Low Risk)**
- Add module-level `setup` to create admin user
- Update helper functions to accept admin user parameter
- Update all `setup` blocks to use shared admin user

**Risk Assessment:** ⚠️ **Low**
- Framework functionality is tested by Ash maintainers
- Business logic tests remain intact
- Admin user sharing maintains test isolation (read-only)
- Consolidation preserves coverage

### Priority 3: Seeds Tests Further Optimization

**Estimated Savings:** 3-5 seconds

**Actions:**
1. Investigate if settings update can be moved to `setup_all`
2. Introduce seeds mode for tests (less data in test mode)
3. Optimize idempotency test (only 1x seeds instead of 2x)

**Risk Assessment:** ⚠️ **Low to Medium**
- Sandbox limitations may prevent `setup_all` usage
- Seeds mode would require careful implementation
- Idempotency test optimization needs to maintain test validity

### Priority 4: Additional Test Isolation Improvements

**Estimated Savings:** Variable (depends on specific tests)

**Actions:**
1. Review tests that load all records (similar to the critical test fix)
2. Add query filters where appropriate
3. Ensure proper test isolation in async tests

**Risk Assessment:** ⚠️ **Very Low**
- Similar to the critical test optimization (proven approach)
- Improves test isolation and reliability

---

## Estimated Total Optimization Potential

| Priority | Optimization | Estimated Savings |
|----------|-------------|-------------------|
| 1 | User LiveView Tests | 14-22s |
| 2 | Policy Tests | 5.5-9s |
| 3 | Seeds Tests Further | 3-5s |
| 4 | Additional Isolation | Variable |
| **Total Potential** | | **22.5-36 seconds** |

**Projected Final Time:** From ~368 seconds (fast suite) to **~332-345 seconds** (~5.5-5.8 minutes) with remaining optimizations

**Note:** Detailed analysis documents available:
- User LiveView Tests: See "Priority 1: User LiveView Tests Optimization" section above
- Policy Tests: See "Priority 2: Policy Tests Optimization" section above

---

## Risk Assessment Summary

### Overall Risk Level: ⚠️ **Low**

All optimizations maintain test coverage while improving performance:

| Optimization | Risk Level | Mitigation |
|-------------|------------|------------|
| Seeds tests reduction | ⚠️ Low | Coverage mapped to domain tests |
| Performance tests tagging | ✅ Very Low | Tests still executed, just separately |
| Critical test optimization | ✅ Very Low | Functionality unchanged, better isolation |
| Future optimizations | ⚠️ Low | Careful implementation with verification |

### Monitoring Plan

#### Success Criteria

- ✅ Seeds tests execute in <20 seconds consistently
- ✅ No increase in seeds-related deployment failures
- ✅ No regression in authorization or membership fee bugs
- ✅ Top 20 slowest tests: < 60 seconds (currently ~44s)
- ✅ Total execution time (without `:slow`): < 10 minutes (currently 6.1 min)
- ⏳ Slow tests execution time: < 2 minutes (currently ~1.3 min)

#### What to Watch For

1. **Production Seeds Failures:**
   - Monitor deployment logs for seeds errors
   - If failures increase, consider restoring detailed tests

2. **Authorization Bugs After Seeds Changes:**
   - If role/permission bugs appear after seeds modifications
   - May indicate need for more seeds-specific role validation

3. **Test Performance Regression:**
   - Monitor test execution times in CI
   - Alert if times increase significantly

4. **Developer Feedback:**
   - If developers report missing test coverage
   - Adjust based on real-world experience

---

## Benchmarking and Analysis

### How to Benchmark Tests

**ExUnit Built-in Benchmarking:**

The test suite is configured to show the slowest tests automatically:

```elixir
# test/test_helper.exs
ExUnit.start(
  slowest: 10  # Shows 10 slowest tests at the end of test run
)
```

**Run Benchmark Analysis:**

```bash
# Show slowest tests
mix test --slowest 20

# Exclude slow tests for faster feedback
mix test --exclude slow --slowest 20

# Run only slow tests
mix test --only slow --slowest 10

# Benchmark specific test file
mix test test/mv_web/member_live/index_member_fields_display_test.exs --slowest 5
```

### Benchmarking Best Practices

1. **Run benchmarks regularly** (e.g., monthly) to catch performance regressions
2. **Compare isolated vs. full runs** to identify test isolation issues
3. **Monitor CI execution times** to track trends over time
4. **Document significant changes** in test performance

---

## Test Suite Structure

### Test Execution Modes

**Fast Tests (Default):**
- Excludes slow tests (`@tag :slow`)
- Used for standard development workflow
- Execution time: ~6 minutes
- Command: `mix test --exclude slow` or `just test-fast`

**Slow Tests:**
- Tests tagged with `@tag :slow` or `@describetag :slow` (25 tests)
- Low risk, >1 second execution time
- UI/Display tests, workflow details, edge cases, performance tests
- Execution time: ~1.3 minutes
- Command: `mix test --only slow` or `just test-slow`
- Excluded from standard CI runs

**Full Test Suite (via Promotion):**
- Triggered by promoting a build to `production` in Drone CI
- Runs all tests (`mix test`) for comprehensive coverage
- Execution time: ~7.4 minutes
- Required before merging to `main` (enforced via branch protection)

**All Tests:**
- Includes both fast and slow tests
- Used for comprehensive validation (pre-merge)
- Execution time: ~7.4 minutes
- Command: `mix test` or `just test`

### Test Organization

Tests are organized to mirror the `lib/` directory structure:

```
test/
├── accounts/              # Accounts domain tests
├── membership/            # Membership domain tests
├── membership_fees/      # Membership fees domain tests
├── mv/                    # Core application tests
│   ├── accounts/         # User-related tests
│   ├── membership/       # Member-related tests
│   └── authorization/    # Authorization tests
├── mv_web/               # Web layer tests
│   ├── controllers/     # Controller tests
│   ├── live/             # LiveView tests
│   └── components/       # Component tests
└── support/              # Test helpers
    ├── conn_case.ex      # Controller test setup
    └── data_case.ex      # Database test setup
```

---

## Best Practices for Test Performance

### When Writing New Tests

1. **Use `async: true`** when possible (for parallel execution)
2. **Filter queries** to only load necessary data
3. **Share fixtures** in `setup_all` when appropriate
4. **Tag performance tests** with `@tag :slow` if they use large datasets
5. **Keep test data minimal** - only create what's needed for the test

### When Optimizing Existing Tests

1. **Measure first** - Use `mix test --slowest` to identify bottlenecks
2. **Compare isolated vs. full runs** - Identify test isolation issues
3. **Optimize setup** - Move shared data to `setup_all` where possible
4. **Filter queries** - Only load data needed for the test
5. **Verify coverage** - Ensure optimizations don't reduce test coverage

### Test Tagging Guidelines

#### Tag as `@tag :slow` when:

1. **Performance Tests:**
   - Explicitly testing performance characteristics
   - Using large datasets (50+ records)
   - Testing scalability or query optimization
   - Validating N+1 query prevention

2. **Low-Risk Tests (>1s):**
   - UI/Display/Formatting tests (not critical for every commit)
   - Workflow detail tests (not core functionality)
   - Edge cases with large datasets
   - Show page tests (core functionality covered by Index/Form tests)
   - Non-critical seeds tests (smoke tests, idempotency)

#### Do NOT tag as `@tag :slow` when:

- ❌ Test is slow due to inefficient setup (fix the setup instead)
- ❌ Test is slow due to bugs (fix the bug instead)
- ❌ Core CRUD operations (Member/User Create/Update/Destroy)
- ❌ Basic Authentication/Authorization
- ❌ Critical Bootstrap (Admin user, system roles)
- ❌ Email Synchronization
- ❌ Representative Policy tests (one per Permission Set + Action)
- ❌ It's an integration test (use `@tag :integration` instead)

---

## Changelog

### 2026-01-28: Initial Optimization Phase

**Completed:**
- ✅ Reduced seeds tests from 13 to 4 tests
- ✅ Tagged 9 performance tests with `@tag :slow`
- ✅ Optimized critical test with query filtering
- ✅ Created slow test suite infrastructure
- ✅ Updated CI/CD to exclude slow tests from standard runs
- ✅ Added promotion-based full test suite pipeline (`check-full`)

**Time Saved:** ~21-30 seconds per test run

### 2026-01-28: Full Test Suite via Promotion Implementation

**Completed:**
- ✅ Analyzed all tests for full test suite candidates
- ✅ Identified 36 tests with low risk and >1s execution time
- ✅ Tagged 25 tests with `@tag :slow` for full test suite (via promotion)
- ✅ Categorized tests by risk level and execution time
- ✅ Documented tagging criteria and guidelines

**Tests Tagged:**
- 2 Seeds tests (non-critical) - 18.1s
- 3 UserLive.ShowTest tests - 10.8s
- 5 UserLive.IndexTest tests - 25.0s
- 3 MemberLive.IndexCustomFieldsDisplayTest tests - 4.9s
- 3 MemberLive.IndexCustomFieldsEdgeCasesTest tests - 3.6s
- 7 RoleLive tests - 7.7s
- 1 MemberAvailableForLinkingTest - 1.5s
- 1 Performance test (already tagged) - 3.8s

**Time Saved:** ~77 seconds per test run

**Total Optimization Impact:**
- **Before:** ~445 seconds (7.4 minutes)
- **After (fast suite):** ~368 seconds (6.1 minutes)
- **Time saved:** ~77 seconds (17% reduction)

**Next Steps:**
- ⏳ Monitor full test suite execution via promotion in CI
- ⏳ Optimize remaining slow tests (Policy tests, etc.)
- ⏳ Further optimize Seeds tests (Priority 3)

---

## References

- **Testing Standards:** `CODE_GUIDELINES.md` - Section 4 (Testing Standards)
- **CI/CD Configuration:** `.drone.yml`
- **Test Helper:** `test/test_helper.exs`
- **Justfile Commands:** `Justfile` (test-fast, test-slow, test-all)

---

## Questions & Answers

**Q: What if seeds create wrong data and break the system?**  
A: The smoke test will fail if seeds raise errors. Domain tests ensure business logic is correct regardless of seeds content.

**Q: What if we add a new critical bootstrap requirement?**  
A: Add a new test to the "Critical bootstrap invariants" section in `test/seeds_test.exs`.

**Q: How do we know the removed tests aren't needed?**  
A: Monitor for 2-3 months. If no seeds-related bugs appear that would have been caught by removed tests, they were redundant.

**Q: Should we restore the tests for important releases?**  
A: Consider running the full test suite (including slow tests) before major releases. Daily development uses the optimized suite.

**Q: How do I add a new performance test?**  
A: Tag it with `@tag :slow` for individual tests or `@describetag :slow` for describe blocks. Use `@describetag` instead of `@moduletag` to avoid tagging unrelated tests. Include measurable performance assertions (query counts, timing with tolerance, etc.). See "Performance Test Guidelines" section above.

**Q: Can I run slow tests locally?**  
A: Yes, use `just test-slow` or `mix test --only slow`. They're excluded from standard runs for faster feedback.

**Q: What is the "full test suite"?**  
A: The full test suite runs **all tests** (`mix test`), including slow and UI tests. Tests tagged with `@tag :slow` or `@describetag :slow` are excluded from standard CI runs (`check-fast`) for faster feedback, but are included when promoting a build to `production` (`check-full`) before merging to `main`.

**Q: Which tests should I tag as `:slow`?**  
A: Tag tests with `@tag :slow` if they: (1) take >1 second, (2) have low risk (not critical for catching regressions), and (3) test UI/Display/Formatting or workflow details. See "Test Tagging Guidelines" section for details.

**Q: What if a slow test fails in the full test suite?**  
A: If a test in the full test suite fails, investigate the failure. If it indicates a critical regression, consider moving it back to the fast suite. If it's a flaky test, fix the test itself. The merge will be blocked until all tests pass.