mitgliederverwaltung/docs/test-optimization-seeds.md

Seeds Test Optimization - Documentation

Overview

The seeds test suite was optimized to reduce test execution time while maintaining coverage of critical deployment requirements.

Date: 2026-01-28
Impact: Reduced test execution time by ~10-16 seconds per full test run
Test Count: Reduced from 13 tests to 4 tests
Seeds Executions: Reduced from 8+ to 5 executions per test run

Measured Performance

Before Optimization:

• Seeds executed: ~8-10 times per test run
• Test execution time: 24-30 seconds
• Tests: 13 comprehensive tests

After Optimization:

• Seeds executed: 5 times (4 tests + 1 for idempotency check)
• Test execution time: 13-17 seconds
• Tests: 4 focused tests
• Time saved: ~10-16 seconds per test run

---

What Was Changed

Before Optimization

• Total Tests: 13 tests across 3 describe blocks
• Execution Pattern: Seeds executed 8+ times per test run
• Execution Time: ~24-30 seconds for seeds tests alone
• Coverage: Extensive validation of seeds output, including:
  • Member/fee type distribution
  • Cycle status variations
  • Detailed role configurations
  • Permission set validations
  • Role assignment behavior

After Optimization

• Total Tests: 4 tests across 2 describe blocks
• Execution Pattern: Seeds executed 4 times (once per test due to sandbox isolation), plus 1 additional time for idempotency test = 5 total executions
• Execution Time: ~17 seconds for seeds tests (down from 24-30 seconds)
• Coverage: Focused on critical deployment requirements:
  • Seeds run without errors (smoke test)
  • Seeds are idempotent
  • Admin user has Admin role
  • System role "Mitglied" exists and is configured correctly

Note on Sandbox Limitation:
Initially, the goal was to run seeds only once using setup_all. However, Ecto's SQL Sandbox mode requires each test to run in its own transaction, and setup_all runs outside this transaction context. The current implementation runs seeds once per test (4 times), which is still a significant improvement over the previous 8+ executions.

Potential Future Optimization:
If further optimization is needed, consider using :shared sandbox mode for seeds tests, which would allow true setup_all usage. However, this adds complexity and potential race conditions.

---

Removed Tests and Coverage Mapping

1. Member/Fee Type Distribution Tests

Removed Tests:

• "at least one member has no membership fee type assigned"
• "each membership fee type has at least one member"

Why Removed:

• These tests validate business logic, not deployment requirements
• Seeds can change (different sample data) without breaking the system

Alternative Coverage:

• Domain tests in test/membership_fees/membership_fee_type_test.exs
• Integration tests in test/membership_fees/membership_fee_type_integration_test.exs
• Business logic: Members can exist with or without fee types (tested in domain)

Risk Assessment: ⚠️ Low Risk

• If seeds fail to create proper fee type assignments, the system still works
• Domain tests ensure the business logic is correct
• UI tests verify the display of fee types

---

2. Cycle Status Variation Tests

Removed Test:

• "members with fee types have cycles with various statuses"

Why Removed:

• Tests cycle generation logic, which is covered by dedicated tests
• Seeds-specific cycle statuses are implementation details, not requirements

Alternative Coverage:

• test/mv/membership_fees/cycle_generator_test.exs - comprehensive cycle generation tests
• test/mv/membership_fees/cycle_generator_edge_cases_test.exs - edge cases
• test/membership_fees/membership_fee_cycle_test.exs - cycle CRUD operations

Risk Assessment: ⚠️ Low Risk

• Cycle generation is thoroughly tested in domain tests
• Seeds could have all paid, all unpaid, or mixed cycles - system works regardless
• Business logic ensures cycles are generated correctly when fee types are assigned

---

3. Detailed Role Configuration Tests

Removed Tests:

• "creates all 5 authorization roles with correct permission sets" (detailed validation)
• "all roles have valid permission_set_names"

Why Removed:

• Detailed role configurations are validated in authorization tests
• Seeds tests should verify bootstrap succeeds, not enumerate all roles

Alternative Coverage:

• test/mv/authorization/role_test.exs - role CRUD and validation
• test/mv/authorization/permission_sets_test.exs - permission set definitions
• test/mv/authorization/checks/has_permission_test.exs - permission logic

Retained Coverage:

• Admin user has Admin role (critical for initial login)
• Mitglied system role exists (critical for default role assignment)

Risk Assessment: ⚠️ Very Low Risk

• Authorization is one of the most thoroughly tested domains
• Policy tests verify each role's permissions in detail
• Seeds tests now focus on "can the system start" not "are all roles perfect"

---

4. Role Assignment Idempotency Tests

Removed Tests:

• "does not change role of users who already have a role"
• "role creation is idempotent" (detailed version)

Why Removed:

• Merged into the general idempotency test
• Specific role assignment behavior is tested in authorization domain

Alternative Coverage:

• General idempotency test covers role creation
• test/mv/accounts/user_test.exs - user role assignment
• Authorization tests cover role assignment logic

Risk Assessment: ⚠️ Very Low Risk

• General idempotency test ensures no duplication
• Authorization tests verify role assignment correctness

---

What Remains

Critical Path Tests (4 tests)

1. Smoke Test: "runs successfully and creates basic data"

   • Verifies seeds complete without errors
   • Confirms basic data structures exist (users, members, custom_fields, roles)
   • Critical for: Initial deployment validation

2. Idempotency Test: "is idempotent when run multiple times"

   • Verifies seeds can be run multiple times without duplicating data
   • Critical for: Re-deployments and zero-downtime deployments

3. Admin Bootstrap: "Admin user has Admin role and can authenticate"

   • Verifies admin user exists with correct role and permissions
   • Critical for: Initial system access

4. System Role Bootstrap: "Mitglied system role exists and is correctly configured"

   • Verifies default user role exists and is marked as system role
   • Critical for: New user registration

---

Future Integration Test Suite

If additional coverage is desired, the removed tests could be moved to a nightly/weekly integration test suite:

Proposed: test/integration/seeds_detailed_test.exs

@moduletag :integration
@moduletag :slow

describe "Seeds data distribution" do
  test "creates diverse sample data for realistic testing"
  test "fee types have realistic member distribution"
  test "cycles have varied statuses for testing"
end

describe "Seeds role configuration" do
  test "creates all 5 authorization roles with correct configs"
  test "all roles have valid permission_set_names from PermissionSets module"
end

When to Run:

• Nightly CI builds
• Before releases
• After seeds modifications
• On-demand: mix test --only integration

---

Monitoring Recommendations

What to Watch For

1. Seeds Failures in Production Deployments

   • If seeds start failing in deployment, restore detailed tests
   • Monitor deployment logs for seeds errors

2. Authorization Bugs After Seeds Changes

   • If role/permission bugs appear after seeds modifications
   • Consider restoring role configuration tests

3. Sample Data Quality

   • If QA team reports unrealistic sample data
   • Consider restoring data distribution tests for integration suite

Success Metrics

• ✅ Test execution time reduced by 15-20 seconds
• ✅ Seeds tests still catch deployment-critical failures
• ✅ No increase in production seeds failures
• ✅ Domain tests continue to provide adequate coverage

---

Rollback Plan

If issues arise, the previous comprehensive tests can be restored from git history:

# View previous version
git show HEAD~1:test/seeds_test.exs

# Restore if needed
git checkout HEAD~1 -- test/seeds_test.exs

Commit with comprehensive tests: (see git log for exact commit hash)

---

Questions & Answers

Q: What if seeds create wrong data and break the system?
A: The smoke test will fail if seeds raise errors. Domain tests ensure business logic is correct regardless of seeds content.

Q: What if we add a new critical bootstrap requirement?
A: Add a new test to the "Critical bootstrap invariants" section.

Q: How do we know the removed tests aren't needed?
A: Monitor for 2-3 months. If no seeds-related bugs appear that would have been caught by removed tests, they were redundant.

Q: Should we restore the tests for important releases?
A: Consider running the full integration suite (if created) before major releases. Daily development uses the minimal suite.

---

Related Documentation

• CODE_GUIDELINES.md - Testing standards and best practices
• docs/roles-and-permissions-architecture.md - Authorization system design
• test/mv/authorization/ - Authorization domain tests
• test/membership_fees/ - Membership fee domain tests