mitgliederverwaltung/docs/test-performance-optimization.md

Test Performance Optimization

Last Updated: 2026-01-28
Status: ✅ Active optimization program

---

Executive Summary

This document provides a comprehensive overview of test performance optimizations, risk assessments, and future opportunities. The test suite execution time has been reduced through systematic analysis and targeted optimizations.

Current Performance Metrics

Metric	Value
Total Execution Time (without :slow tests)	~614 seconds (~10.2 minutes)
Total Tests	1,368 tests (+ 25 doctests)
Async Execution	317.7 seconds
Sync Execution	296.2 seconds
Slow Tests Excluded	9 tests (tagged with @tag :slow)
Top 20 Slowest Tests	81.2 seconds (13.2% of total time)

Optimization Impact Summary

Optimization	Tests Affected	Time Saved	Status
Seeds tests reduction	13 → 4 tests	~10-16s	✅ Completed
Performance tests tagging	9 tests	~3-4s per run	✅ Completed
Critical test query filtering	1 test	~8-10s	✅ Completed
Total Saved		~21-30s

---

Completed Optimizations

1. Seeds Test Suite Optimization

Date: 2026-01-28
Status: ✅ Completed

What Changed

• Reduced test count: From 13 tests to 4 tests (69% reduction)
• Reduced seeds executions: From 8-10 times to 5 times per test run
• Execution time: From 24-30 seconds to 13-17 seconds
• Time saved: ~10-16 seconds per test run (40-50% faster)

Removed Tests (9 tests)

Tests were removed because their functionality is covered by domain-specific test suites:

1. "at least one member has no membership fee type assigned" → Covered by membership_fees/*_test.exs
2. "each membership fee type has at least one member" → Covered by membership_fees/*_test.exs
3. "members with fee types have cycles with various statuses" → Covered by cycle_generator_test.exs
4. "creates all 5 authorization roles with correct permission sets" → Covered by authorization/*_test.exs
5. "all roles have valid permission_set_names" → Covered by authorization/permission_sets_test.exs
6. "does not change role of users who already have a role" → Merged into idempotency test
7. "role creation is idempotent" (detailed) → Merged into general idempotency test

Retained Tests (4 tests)

Critical deployment requirements are still covered:

1. ✅ Smoke Test: Seeds run successfully and create basic data
2. ✅ Idempotency Test: Seeds can be run multiple times without duplicating data
3. ✅ Admin Bootstrap: Admin user exists with Admin role (critical for initial access)
4. ✅ System Role Bootstrap: Mitglied system role exists (critical for user registration)

Risk Assessment

Removed Test Category	Alternative Coverage	Risk Level
Member/fee type distribution	membership_fees/*_test.exs	⚠️ Low
Cycle status variations	cycle_generator_test.exs	⚠️ Low
Detailed role configs	authorization/*_test.exs	⚠️ Very Low
Permission set validation	permission_sets_test.exs	⚠️ Very Low

Overall Risk: ⚠️ Low - All removed tests have equivalent or better coverage in domain-specific test suites.

---

2. Performance Test Suite Separation

Date: 2026-01-28
Status: ✅ Completed

What Changed

Performance tests that explicitly validate performance characteristics are now tagged with @tag :slow or @moduletag :slow and excluded from standard test runs.

Identified Performance Tests (9 tests)

1. Member LiveView - Boolean Filter Performance (test/mv_web/member_live/index_test.exs)

   • Test: "boolean filter performance with 150 members"
   • Duration: ~3.8 seconds
   • Creates 150 members to test filter performance

2. Group LiveView - Index Performance (test/mv_web/live/group_live/index_test.exs)

   • 2 tests validating efficient page loading and member count calculation

3. Group LiveView - Show Performance (test/mv_web/live/group_live/show_test.exs)

   • 2 tests validating efficient member list loading and slug lookup

4. Member LiveView - Membership Fee Status Performance (test/mv_web/member_live/index_membership_fee_status_test.exs)

   • 1 test validating efficient cycle loading without N+1 queries

5. Group Integration - Query Performance (test/membership/group_integration_test.exs)

   • 1 test validating N+1 query prevention for group-member relationships

Execution Commands

Fast Tests (Default):

just test-fast
# or
mix test --exclude slow

Performance Tests Only:

just test-slow
# or
mix test --only slow

All Tests:

just test
# or
mix test

CI/CD Integration

• Standard CI: Runs mix test --exclude slow for faster feedback loops
• Nightly Builds: Separate pipeline (nightly-tests) runs daily at 2 AM and executes all performance tests
• Manual Execution: Can be triggered via Drone CLI or Web UI

Risk Assessment

Risk Level: ✅ Very Low

• Performance tests are still executed, just separately
• Standard test runs are faster, improving developer feedback
• Nightly builds ensure comprehensive coverage
• No functionality is lost, only execution timing changed

---

3. Critical Test Optimization

Date: 2026-01-28
Status: ✅ Completed

Problem Identified

The test test respects show_in_overview config was the slowest test in the suite:

• Isolated execution: 4.8 seconds
• In full test run: 14.7 seconds
• Difference: 9.9 seconds (test isolation issue)

Root Cause

The test loaded all members from the database, not just the 2 members from the test setup. In full test runs, many members from other tests were present in the database, significantly slowing down the query.

Solution Implemented

Query Filtering: Added search query parameter to filter to only the expected member.

Code Change:

# Before:
{:ok, _view, html} = live(conn, "/members")

# After:
{:ok, _view, html} = live(conn, "/members?query=Alice")

Results

Execution	Before	After	Improvement
Isolated	4.8s	1.1s	-77% (3.7s saved)
In Module	4.2s	0.4s	-90% (3.8s saved)
Expected in Full Run	14.7s	~4-6s	-65% to -73% (8-10s saved)

Risk Assessment

Risk Level: ✅ Very Low

• Test functionality unchanged - only loads expected data
• All assertions still pass
• Test is now faster and more isolated
• No impact on test coverage

---

Current Performance Analysis

Top 20 Slowest Tests (without :slow)

Rank	Test	File	Time	% of Top 20
1	test respects show_in_overview config	index_member_fields_display_test.exs	~4-6s (optimized)	~6-7%
2	test Seeds script is idempotent when run multiple times	seeds_test.exs	5.0s	6.2%
3	test sorting functionality initially sorts by email ascending	user_live/index_test.exs	4.5s	5.5%
4	test Seeds script runs successfully and creates basic data	seeds_test.exs	4.3s	5.3%
5-20	Various User LiveView and Policy tests	Multiple files	2.6-4.2s each	3-5% each

Total Top 20: ~81 seconds (13.2% of total time)

Performance Hotspots Identified

1. Seeds Tests (~16.2s for 4 tests)

Status: ✅ Optimized (reduced from 13 tests)
Remaining Optimization Potential: 3-5 seconds

Opportunities:

• Settings update could potentially be moved to setup_all (if sandbox allows)
• Seeds execution could be further optimized (less data in test mode)
• Idempotency test could be optimized (only 1x seeds instead of 2x)

2. User LiveView Tests (~35.5s for 10 tests)

Status: ⏳ Identified for optimization
Optimization Potential: 15-20 seconds

Files:

• test/mv_web/user_live/index_test.exs (3 tests, ~10.2s)
• test/mv_web/user_live/form_test.exs (4 tests, ~15.0s)
• test/mv_web/user_live/show_test.exs (3 tests, ~10.3s)

Patterns:

• Many tests create user/member data
• LiveView mounts are expensive
• Form submissions with validations are slow

Recommended Actions:

• Move shared fixtures to setup_all
• Reduce test data volume (3-5 users instead of 10+)
• Optimize setup patterns for recurring patterns

3. Policy Tests (~8.7s for 3 tests)

Status: ⏳ Identified for optimization
Optimization Potential: 5-8 seconds

Files:

• test/mv/membership/member_policies_test.exs (2 tests, ~6.1s)
• test/mv/accounts/user_policies_test.exs (1 test, ~2.6s)

Pattern:

• Each test creates new roles/users/members
• Roles are identical across tests

Recommended Actions:

• Create roles in setup_all (shared across tests)
• Reuse common fixtures
• Maintain test isolation while optimizing setup

---

Future Optimization Opportunities

Priority 1: User LiveView Tests Optimization

Estimated Savings: 15-20 seconds

Actions:

1. Move shared fixtures to setup_all where possible
2. Reduce test data volume (use 3-5 users instead of 10+)
3. Analyze and optimize recurring setup patterns
4. Consider mocking expensive operations (if appropriate)

Risk Assessment: ⚠️ Low

• Requires careful testing to ensure isolation is maintained
• Should verify that shared fixtures don't cause test interdependencies

Priority 2: Policy Tests Optimization

Estimated Savings: 5-8 seconds

Actions:

1. Create roles in setup_all (they're identical across tests)
2. Reuse common fixtures (users, members)
3. Maintain test isolation while optimizing setup

Risk Assessment: ⚠️ Low

• Roles are read-only in tests, safe to share
• Need to ensure user/member fixtures don't interfere with each other

Priority 3: Seeds Tests Further Optimization

Estimated Savings: 3-5 seconds

Actions:

1. Investigate if settings update can be moved to setup_all
2. Introduce seeds mode for tests (less data in test mode)
3. Optimize idempotency test (only 1x seeds instead of 2x)

Risk Assessment: ⚠️ Low to Medium

• Sandbox limitations may prevent setup_all usage
• Seeds mode would require careful implementation
• Idempotency test optimization needs to maintain test validity

Priority 4: Additional Test Isolation Improvements

Estimated Savings: Variable (depends on specific tests)

Actions:

1. Review tests that load all records (similar to the critical test fix)
2. Add query filters where appropriate
3. Ensure proper test isolation in async tests

Risk Assessment: ⚠️ Very Low

• Similar to the critical test optimization (proven approach)
• Improves test isolation and reliability

---

Estimated Total Optimization Potential

Priority	Optimization	Estimated Savings
1	User LiveView Tests	15-20s
2	Policy Tests	5-8s
3	Seeds Tests Further	3-5s
4	Additional Isolation	Variable
Total Potential		23-33 seconds

Projected Final Time: From ~614 seconds to ~580-590 seconds (~9.5-10 minutes)

---

Risk Assessment Summary

Overall Risk Level: ⚠️ Low

All optimizations maintain test coverage while improving performance:

Optimization	Risk Level	Mitigation
Seeds tests reduction	⚠️ Low	Coverage mapped to domain tests
Performance tests tagging	✅ Very Low	Tests still executed, just separately
Critical test optimization	✅ Very Low	Functionality unchanged, better isolation
Future optimizations	⚠️ Low	Careful implementation with verification

Monitoring Plan

Success Criteria

• ✅ Seeds tests execute in <20 seconds consistently
• ✅ No increase in seeds-related deployment failures
• ✅ No regression in authorization or membership fee bugs
• ⏳ Top 20 slowest tests: < 60 seconds (currently 81.2s)
• ⏳ Total execution time (without :slow): < 10 minutes (currently 10.2 min)

What to Watch For

1. Production Seeds Failures:

   • Monitor deployment logs for seeds errors
   • If failures increase, consider restoring detailed tests

2. Authorization Bugs After Seeds Changes:

   • If role/permission bugs appear after seeds modifications
   • May indicate need for more seeds-specific role validation

3. Test Performance Regression:

   • Monitor test execution times in CI
   • Alert if times increase significantly

4. Developer Feedback:

   • If developers report missing test coverage
   • Adjust based on real-world experience

---

Benchmarking and Analysis

How to Benchmark Tests

ExUnit Built-in Benchmarking:

The test suite is configured to show the slowest tests automatically:

# test/test_helper.exs
ExUnit.start(
  slowest: 10  # Shows 10 slowest tests at the end of test run
)

Run Benchmark Analysis:

# Show slowest tests
mix test --slowest 20

# Exclude slow tests for faster feedback
mix test --exclude slow --slowest 20

# Run only slow tests
mix test --only slow --slowest 10

# Benchmark specific test file
mix test test/mv_web/member_live/index_member_fields_display_test.exs --slowest 5

Benchmarking Best Practices

1. Run benchmarks regularly (e.g., monthly) to catch performance regressions
2. Compare isolated vs. full runs to identify test isolation issues
3. Monitor CI execution times to track trends over time
4. Document significant changes in test performance

---

Test Suite Structure

Test Execution Modes

Fast Tests (Default):

• Excludes performance tests (@tag :slow)
• Used for standard development workflow
• Command: mix test --exclude slow or just test-fast

Performance Tests:

• Explicitly tagged performance tests
• Run separately or in nightly builds
• Command: mix test --only slow or just test-slow

All Tests:

• Includes both fast and slow tests
• Used for comprehensive validation
• Command: mix test or just test

Test Organization

Tests are organized to mirror the lib/ directory structure:

test/
├── accounts/              # Accounts domain tests
├── membership/            # Membership domain tests
├── membership_fees/      # Membership fees domain tests
├── mv/                    # Core application tests
│   ├── accounts/         # User-related tests
│   ├── membership/       # Member-related tests
│   └── authorization/    # Authorization tests
├── mv_web/               # Web layer tests
│   ├── controllers/     # Controller tests
│   ├── live/             # LiveView tests
│   └── components/       # Component tests
└── support/              # Test helpers
    ├── conn_case.ex      # Controller test setup
    └── data_case.ex      # Database test setup

---

Best Practices for Test Performance

When Writing New Tests

1. Use async: true when possible (for parallel execution)
2. Filter queries to only load necessary data
3. Share fixtures in setup_all when appropriate
4. Tag performance tests with @tag :slow if they use large datasets
5. Keep test data minimal - only create what's needed for the test

When Optimizing Existing Tests

1. Measure first - Use mix test --slowest to identify bottlenecks
2. Compare isolated vs. full runs - Identify test isolation issues
3. Optimize setup - Move shared data to setup_all where possible
4. Filter queries - Only load data needed for the test
5. Verify coverage - Ensure optimizations don't reduce test coverage

Performance Test Guidelines

Tag as :slow when:

• Explicitly testing performance characteristics
• Using large datasets (50+ records)
• Testing scalability or query optimization
• Validating N+1 query prevention

Do NOT tag as :slow when:

• Test is slow due to inefficient setup (fix the setup instead)
• Test is slow due to bugs (fix the bug instead)
• It's an integration test (use @tag :integration instead)

---

Changelog

2026-01-28: Initial Optimization Phase

Completed:

• ✅ Reduced seeds tests from 13 to 4 tests
• ✅ Tagged 9 performance tests with @tag :slow
• ✅ Optimized critical test with query filtering
• ✅ Created slow test suite infrastructure
• ✅ Updated CI/CD to exclude slow tests from standard runs
• ✅ Added nightly CI pipeline for slow tests

Time Saved: ~21-30 seconds per test run

Next Steps:

• ⏳ Optimize User LiveView tests (Priority 1)
• ⏳ Optimize Policy tests (Priority 2)
• ⏳ Further optimize Seeds tests (Priority 3)

---

References

• Testing Standards: CODE_GUIDELINES.md - Section 4 (Testing Standards)
• CI/CD Configuration: .drone.yml
• Test Helper: test/test_helper.exs
• Justfile Commands: Justfile (test-fast, test-slow, test-all)

---

Questions & Answers

Q: What if seeds create wrong data and break the system?
A: The smoke test will fail if seeds raise errors. Domain tests ensure business logic is correct regardless of seeds content.

Q: What if we add a new critical bootstrap requirement?
A: Add a new test to the "Critical bootstrap invariants" section in test/seeds_test.exs.

Q: How do we know the removed tests aren't needed?
A: Monitor for 2-3 months. If no seeds-related bugs appear that would have been caught by removed tests, they were redundant.

Q: Should we restore the tests for important releases?
A: Consider running the full test suite (including slow tests) before major releases. Daily development uses the optimized suite.

Q: How do I add a new performance test?
A: Tag it with @tag :slow or @moduletag :slow. See "Performance Test Guidelines" section above.

Q: Can I run slow tests locally?
A: Yes, use just test-slow or mix test --only slow. They're excluded from standard runs for faster feedback.