mitgliederverwaltung/docs/test-optimization-summary.md

Test Performance Optimization - Summary

Date: 2026-01-28
Status: ✅ Completed (Phase 1 - Seeds Tests)

---

Executive Summary

The seeds test suite was optimized to reduce redundant test execution while maintaining critical deployment coverage. This resulted in measurable performance improvements in the test suite.

Key Metrics

Metric	Before	After	Improvement
Test Count	13 tests	4 tests	-9 tests (69% reduction)
Seeds Executions	8-10 times	5 times	-3 to -5 executions (40-60% reduction)
Execution Time	24-30 seconds	13-17 seconds	~10-16 seconds saved (40-50% faster)
Time per Test Run	~2.3s average	~3.4s average	Slightly slower per test, but fewer tests

Overall Impact

• Daily Time Savings: For a team running tests 50 times/day: ~12 minutes saved per day
• Annual Time Savings: ~75 hours saved per year (based on 250 working days)
• Developer Experience: Faster feedback loop during development

---

What Changed

Removed (9 tests):

1. "at least one member has no membership fee type assigned" - Business logic, not bootstrap
2. "each membership fee type has at least one member" - Sample data validation
3. "members with fee types have cycles with various statuses" - Cycle generation logic
4. "creates all 5 authorization roles with correct permission sets" (detailed) - Enumeration test
5. "all roles have valid permission_set_names" - Covered by authorization tests
6. "does not change role of users who already have a role" - Merged into idempotency
7. "role creation is idempotent" (detailed) - Merged into general idempotency test

Retained (4 tests):

1. ✅ Smoke Test: Seeds run successfully and create basic data
2. ✅ Idempotency Test: Seeds can be run multiple times without duplicating data
3. ✅ Admin Bootstrap: Admin user exists with Admin role (critical for initial access)
4. ✅ System Role Bootstrap: Mitglied system role exists (critical for user registration)

---

Risk Assessment

Coverage Gaps Analysis

Removed Test	Alternative Coverage	Risk Level
Member/fee type distribution	membership_fees/*_test.exs	⚠️ Low
Cycle status variations	cycle_generator_test.exs	⚠️ Low
Detailed role configs	authorization/*_test.exs	⚠️ Very Low
Permission set validation	permission_sets_test.exs	⚠️ Very Low

Overall Risk: ⚠️ Low - All removed tests have equivalent or better coverage in domain-specific test suites.

---

Verification

Test Run Output

# Before optimization
$ mix test test/seeds_test.exs
Finished in 24.3 seconds
13 tests, 0 failures

# After optimization
$ mix test test/seeds_test.exs  
Finished in 13.6 seconds
4 tests, 0 failures

Top Slowest Tests (After Optimization)

1. "is idempotent when run multiple times" - 5.5s (includes extra seeds run)
2. "Mitglied system role exists" - 2.6s
3. "runs successfully and creates basic data" - 2.6s
4. "Admin user has Admin role" - 2.6s

---

Next Steps

Additional Optimization Opportunities

Based on the initial analysis (mix test --slowest 20), the following test files are candidates for optimization:

1. test/mv_web/member_live/index_member_fields_display_test.exs (~10.3s for single test)

   • Large data setup
   • Multiple LiveView renders
   • Potential savings: 5-8 seconds

2. test/mv_web/user_live/index_test.exs (~10s total for multiple tests)

   • Complex sorting/filtering tests
   • Repeated user creation
   • Potential savings: 3-5 seconds

3. test/mv/membership/member_policies_test.exs (~20s cumulative)

   • Many similar policy tests
   • Role/user creation in each test
   • Potential savings: 5-10 seconds (via shared fixtures)

4. Performance tests (~3.8s for single test with 150 members)

   • Mark with @tag :slow
   • Run separately or in nightly builds
   • Potential savings: 3-4 seconds in standard runs

Recommended Actions

1. ✅ Done: Optimize seeds tests (this document)
2. ⏳ Next: Review and optimize LiveView tests with large data setups
3. ⏳ Next: Implement shared fixtures for policy tests
4. ✅ Done: Tag performance tests as :slow for conditional execution
5. ✅ Done: Nightly integration test suite for comprehensive coverage

---

Slow Test Suite

Performance tests have been tagged with @tag :slow and separated into a dedicated test suite.

Structure

• Performance Tests: Explicit tests that validate performance characteristics (e.g., N+1 query prevention, filter performance with large datasets)
• Tagging: All performance tests are tagged with @tag :slow or @moduletag :slow
• Execution: Standard test runs exclude slow tests, but they can be executed on demand

Execution

Fast Tests (Default):

just test-fast
# or
mix test --exclude slow

# With specific files or options
just test-fast test/membership/member_test.exs
just test-fast --seed 123

Performance Tests Only:

just test-slow
# or
mix test --only slow

# With specific files or options
just test-slow test/mv_web/member_live/index_test.exs
just test-slow --seed 123

All Tests:

just test
# or
mix test

# With specific files or options
just test-all test/mv_web/
just test-all --max-failures 5

Note: All suite commands (test-fast, test-slow, test-all) support additional arguments. The suite semantics (tags) are always preserved - additional arguments are appended to the command.

CI/CD Integration

• Standard CI: Runs mix test --exclude slow for faster feedback loops
• Nightly Builds: Separate pipeline (nightly-tests) runs daily at 2 AM and executes all performance tests
• Manual Execution:
  • Local: Performance tests can be executed anytime with just test-slow
  • CI: The nightly pipeline can be manually triggered via Drone CLI or Web UI:

    drone build start <owner>/<repo> <branch> --event custom
    

Further Details

See test-slow-suite.md for complete documentation of the Slow Test Suite.

---

Monitoring Plan

Success Criteria (Next 3 Months)

• ✅ Seeds tests execute in <20 seconds consistently
• ✅ No increase in seeds-related deployment failures
• ✅ No regression in authorization or membership fee bugs that would have been caught by removed tests

What to Watch For

1. Production Seeds Failures:

   • Monitor deployment logs for seeds errors
   • If failures increase, consider restoring detailed tests

2. Authorization Bugs After Seeds Changes:

   • If role/permission bugs appear after seeds modifications
   • May indicate need for more seeds-specific role validation

3. Developer Feedback:

   • If developers report missing test coverage
   • Adjust based on real-world experience

---

References

• Detailed Documentation: docs/test-optimization-seeds.md
• Slow Test Suite: docs/test-slow-suite.md
• Test File: test/seeds_test.exs
• Original Analysis: Internal benchmarking session (2026-01-28)
• Related Guidelines: CODE_GUIDELINES.md - Section 4 (Testing Standards)

---

Changelog

2026-01-28: Initial Optimization

• Reduced seeds tests from 13 to 4
• Consolidated setup using per-test seeds execution
• Documented coverage mapping and risk assessment
• Measured time savings: 10-16 seconds per run

---

Appendix: Full Test Output Example

$ mix test test/seeds_test.exs --slowest 10

Mv.SeedsTest [test/seeds_test.exs]
  * test Seeds script is idempotent when run multiple times (5490.6ms)
  * test Critical bootstrap invariants Mitglied system role exists (2630.2ms)
  * test Seeds script runs successfully and creates basic data (2624.4ms)
  * test Critical bootstrap invariants Admin user has Admin role (2619.0ms)

Finished in 13.6 seconds (0.00s async, 13.6s sync)

Top 10 slowest (13.3s), 98.1% of total time:
  * test Seeds script is idempotent when run multiple times (5490.6ms)
  * test Critical bootstrap invariants Mitglied system role exists (2630.2ms)  
  * test Seeds script runs successfully and creates basic data (2624.4ms)
  * test Critical bootstrap invariants Admin user has Admin role (2619.0ms)

4 tests, 0 failures