local-it/mitgliederverwaltung
2
3
Fork 0
Code Issues 66 Pull requests 6 Projects 3 Releases Packages 1 Activity Actions

test: optimize single test and update docs
All checks were successful
continuous-integration/drone/push Build is passing
Details

Browse source
This commit is contained in:
Simon 2026-01-28 13:33:39 +01:00
parent 6efad280bd
commit 15d328afbf
Signed by: simon
GPG key ID: 40E7A58C4AA1EDB2
5 changed files with 546 additions and 820 deletions
Download patch file Download diff file Expand all files Collapse all files

267
docs/test-optimization-seeds.md
View file

@ -1,267 +0,0 @@
# Seeds Test Optimization - Documentation
## Overview
The seeds test suite was optimized to reduce test execution time while maintaining coverage of critical deployment requirements.
**Date:** 2026-01-28
**Impact:** Reduced test execution time by ~10-16 seconds per full test run
**Test Count:** Reduced from 13 tests to 4 tests
**Seeds Executions:** Reduced from 8+ to 5 executions per test run
### Measured Performance
**Before Optimization:**
- Seeds executed: ~8-10 times per test run
- Test execution time: 24-30 seconds
- Tests: 13 comprehensive tests
**After Optimization:**
- Seeds executed: 5 times (4 tests + 1 for idempotency check)
- Test execution time: 13-17 seconds
- Tests: 4 focused tests
- **Time saved:** ~10-16 seconds per test run
---
## What Was Changed
### Before Optimization
- **Total Tests:** 13 tests across 3 describe blocks
- **Execution Pattern:** Seeds executed 8+ times per test run
- **Execution Time:** ~24-30 seconds for seeds tests alone
- **Coverage:** Extensive validation of seeds output, including:
- Member/fee type distribution
- Cycle status variations
- Detailed role configurations
- Permission set validations
- Role assignment behavior
### After Optimization
- **Total Tests:** 4 tests across 2 describe blocks
- **Execution Pattern:** Seeds executed 4 times (once per test due to sandbox isolation), plus 1 additional time for idempotency test = 5 total executions
- **Execution Time:** ~17 seconds for seeds tests (down from 24-30 seconds)
- **Coverage:** Focused on critical deployment requirements:
- Seeds run without errors (smoke test)
- Seeds are idempotent
- Admin user has Admin role
- System role "Mitglied" exists and is configured correctly
**Note on Sandbox Limitation:**
Initially, the goal was to run seeds only once using `setup_all`. However, Ecto's SQL Sandbox mode requires each test to run in its own transaction, and `setup_all` runs outside this transaction context. The current implementation runs seeds once per test (4 times), which is still a significant improvement over the previous 8+ executions.
**Potential Future Optimization:**
If further optimization is needed, consider using `:shared` sandbox mode for seeds tests, which would allow true `setup_all` usage. However, this adds complexity and potential race conditions.
---
## Removed Tests and Coverage Mapping
### 1. Member/Fee Type Distribution Tests
**Removed Tests:**
- `"at least one member has no membership fee type assigned"`
- `"each membership fee type has at least one member"`
**Why Removed:**
- These tests validate business logic, not deployment requirements
- Seeds can change (different sample data) without breaking the system
**Alternative Coverage:**
- Domain tests in `test/membership_fees/membership_fee_type_test.exs`
- Integration tests in `test/membership_fees/membership_fee_type_integration_test.exs`
- Business logic: Members can exist with or without fee types (tested in domain)
**Risk Assessment:** ⚠️ **Low Risk**
- If seeds fail to create proper fee type assignments, the system still works
- Domain tests ensure the business logic is correct
- UI tests verify the display of fee types
---
### 2. Cycle Status Variation Tests
**Removed Test:**
- `"members with fee types have cycles with various statuses"`
**Why Removed:**
- Tests cycle generation logic, which is covered by dedicated tests
- Seeds-specific cycle statuses are implementation details, not requirements
**Alternative Coverage:**
- `test/mv/membership_fees/cycle_generator_test.exs` - comprehensive cycle generation tests
- `test/mv/membership_fees/cycle_generator_edge_cases_test.exs` - edge cases
- `test/membership_fees/membership_fee_cycle_test.exs` - cycle CRUD operations
**Risk Assessment:** ⚠️ **Low Risk**
- Cycle generation is thoroughly tested in domain tests
- Seeds could have all paid, all unpaid, or mixed cycles - system works regardless
- Business logic ensures cycles are generated correctly when fee types are assigned
---
### 3. Detailed Role Configuration Tests
**Removed Tests:**
- `"creates all 5 authorization roles with correct permission sets"` (detailed validation)
- `"all roles have valid permission_set_names"`
**Why Removed:**
- Detailed role configurations are validated in authorization tests
- Seeds tests should verify bootstrap succeeds, not enumerate all roles
**Alternative Coverage:**
- `test/mv/authorization/role_test.exs` - role CRUD and validation
- `test/mv/authorization/permission_sets_test.exs` - permission set definitions
- `test/mv/authorization/checks/has_permission_test.exs` - permission logic
**Retained Coverage:**
- Admin user has Admin role (critical for initial login)
- Mitglied system role exists (critical for default role assignment)
**Risk Assessment:** ⚠️ **Very Low Risk**
- Authorization is one of the most thoroughly tested domains
- Policy tests verify each role's permissions in detail
- Seeds tests now focus on "can the system start" not "are all roles perfect"
---
### 4. Role Assignment Idempotency Tests
**Removed Tests:**
- `"does not change role of users who already have a role"`
- `"role creation is idempotent"` (detailed version)
**Why Removed:**
- Merged into the general idempotency test
- Specific role assignment behavior is tested in authorization domain
**Alternative Coverage:**
- General idempotency test covers role creation
- `test/mv/accounts/user_test.exs` - user role assignment
- Authorization tests cover role assignment logic
**Risk Assessment:** ⚠️ **Very Low Risk**
- General idempotency test ensures no duplication
- Authorization tests verify role assignment correctness
---
## What Remains
### Critical Path Tests (4 tests)
1. **Smoke Test:** "runs successfully and creates basic data"
- Verifies seeds complete without errors
- Confirms basic data structures exist (users, members, custom_fields, roles)
- **Critical for:** Initial deployment validation
2. **Idempotency Test:** "is idempotent when run multiple times"
- Verifies seeds can be run multiple times without duplicating data
- **Critical for:** Re-deployments and zero-downtime deployments
3. **Admin Bootstrap:** "Admin user has Admin role and can authenticate"
- Verifies admin user exists with correct role and permissions
- **Critical for:** Initial system access
4. **System Role Bootstrap:** "Mitglied system role exists and is correctly configured"
- Verifies default user role exists and is marked as system role
- **Critical for:** New user registration
---
## Future Integration Test Suite
If additional coverage is desired, the removed tests could be moved to a nightly/weekly integration test suite:
### Proposed: `test/integration/seeds_detailed_test.exs`
```elixir
@moduletag :integration
@moduletag :slow
describe "Seeds data distribution" do
test "creates diverse sample data for realistic testing"
test "fee types have realistic member distribution"
test "cycles have varied statuses for testing"
end
describe "Seeds role configuration" do
test "creates all 5 authorization roles with correct configs"
test "all roles have valid permission_set_names from PermissionSets module"
end
```
**When to Run:**
- Nightly CI builds
- Before releases
- After seeds modifications
- On-demand: `mix test --only integration`
---
## Monitoring Recommendations
### What to Watch For
1. **Seeds Failures in Production Deployments**
- If seeds start failing in deployment, restore detailed tests
- Monitor deployment logs for seeds errors
2. **Authorization Bugs After Seeds Changes**
- If role/permission bugs appear after seeds modifications
- Consider restoring role configuration tests
3. **Sample Data Quality**
- If QA team reports unrealistic sample data
- Consider restoring data distribution tests for integration suite
### Success Metrics
- ✅ Test execution time reduced by 15-20 seconds
- ✅ Seeds tests still catch deployment-critical failures
- ✅ No increase in production seeds failures
- ✅ Domain tests continue to provide adequate coverage
---
## Rollback Plan
If issues arise, the previous comprehensive tests can be restored from git history:
```bash
# View previous version
git show HEAD~1:test/seeds_test.exs
# Restore if needed
git checkout HEAD~1 -- test/seeds_test.exs
```
**Commit with comprehensive tests:** (see git log for exact commit hash)
---
## Questions & Answers
**Q: What if seeds create wrong data and break the system?**
A: The smoke test will fail if seeds raise errors. Domain tests ensure business logic is correct regardless of seeds content.
**Q: What if we add a new critical bootstrap requirement?**
A: Add a new test to the "Critical bootstrap invariants" section.
**Q: How do we know the removed tests aren't needed?**
A: Monitor for 2-3 months. If no seeds-related bugs appear that would have been caught by removed tests, they were redundant.
**Q: Should we restore the tests for important releases?**
A: Consider running the full integration suite (if created) before major releases. Daily development uses the minimal suite.
---
## Related Documentation
- `CODE_GUIDELINES.md` - Testing standards and best practices
- `docs/roles-and-permissions-architecture.md` - Authorization system design
- `test/mv/authorization/` - Authorization domain tests
- `test/membership_fees/` - Membership fee domain tests

252
docs/test-optimization-summary.md
View file

@ -1,252 +0,0 @@
# Test Performance Optimization - Summary
**Date:** 2026-01-28
**Status:** ✅ Completed (Phase 1 - Seeds Tests)
---
## Executive Summary
The seeds test suite was optimized to reduce redundant test execution while maintaining critical deployment coverage. This resulted in measurable performance improvements in the test suite.
### Key Metrics
| Metric | Before | After | Improvement |
|--------|--------|-------|-------------|
| **Test Count** | 13 tests | 4 tests | -9 tests (69% reduction) |
| **Seeds Executions** | 8-10 times | 5 times | -3 to -5 executions (40-60% reduction) |
| **Execution Time** | 24-30 seconds | 13-17 seconds | **~10-16 seconds saved** (40-50% faster) |
| **Time per Test Run** | ~2.3s average | ~3.4s average | Slightly slower per test, but fewer tests |
### Overall Impact
- **Daily Time Savings:** For a team running tests 50 times/day: **~12 minutes saved per day**
- **Annual Time Savings:** **~75 hours saved per year** (based on 250 working days)
- **Developer Experience:** Faster feedback loop during development
---
## What Changed
### Removed (9 tests):
1. `"at least one member has no membership fee type assigned"` - Business logic, not bootstrap
2. `"each membership fee type has at least one member"` - Sample data validation
3. `"members with fee types have cycles with various statuses"` - Cycle generation logic
4. `"creates all 5 authorization roles with correct permission sets"` (detailed) - Enumeration test
5. `"all roles have valid permission_set_names"` - Covered by authorization tests
6. `"does not change role of users who already have a role"` - Merged into idempotency
7. `"role creation is idempotent"` (detailed) - Merged into general idempotency test
### Retained (4 tests):
1. ✅ **Smoke Test:** Seeds run successfully and create basic data
2. ✅ **Idempotency Test:** Seeds can be run multiple times without duplicating data
3. ✅ **Admin Bootstrap:** Admin user exists with Admin role (critical for initial access)
4. ✅ **System Role Bootstrap:** Mitglied system role exists (critical for user registration)
---
## Risk Assessment
### Coverage Gaps Analysis
| Removed Test | Alternative Coverage | Risk Level |
|--------------|---------------------|------------|
| Member/fee type distribution | `membership_fees/*_test.exs` | ⚠️ Low |
| Cycle status variations | `cycle_generator_test.exs` | ⚠️ Low |
| Detailed role configs | `authorization/*_test.exs` | ⚠️ Very Low |
| Permission set validation | `permission_sets_test.exs` | ⚠️ Very Low |
**Overall Risk:** ⚠️ **Low** - All removed tests have equivalent or better coverage in domain-specific test suites.
---
## Verification
### Test Run Output
```bash
# Before optimization
$ mix test test/seeds_test.exs
Finished in 24.3 seconds
13 tests, 0 failures
# After optimization
$ mix test test/seeds_test.exs
Finished in 13.6 seconds
4 tests, 0 failures
```
### Top Slowest Tests (After Optimization)
1. `"is idempotent when run multiple times"` - 5.5s (includes extra seeds run)
2. `"Mitglied system role exists"` - 2.6s
3. `"runs successfully and creates basic data"` - 2.6s
4. `"Admin user has Admin role"` - 2.6s
---
## Next Steps
### Additional Optimization Opportunities
Based on the initial analysis (`mix test --slowest 20`), the following test files are candidates for optimization:
1. **`test/mv_web/member_live/index_member_fields_display_test.exs`** (~10.3s for single test)
- Large data setup
- Multiple LiveView renders
- **Potential savings:** 5-8 seconds
2. **`test/mv_web/user_live/index_test.exs`** (~10s total for multiple tests)
- Complex sorting/filtering tests
- Repeated user creation
- **Potential savings:** 3-5 seconds
3. **`test/mv/membership/member_policies_test.exs`** (~20s cumulative)
- Many similar policy tests
- Role/user creation in each test
- **Potential savings:** 5-10 seconds (via shared fixtures)
4. **Performance tests** (~3.8s for single test with 150 members)
- Mark with `@tag :slow`
- Run separately or in nightly builds
- **Potential savings:** 3-4 seconds in standard runs
### Recommended Actions
1. ✅ **Done:** Optimize seeds tests (this document)
2. ⏳ **Next:** Review and optimize LiveView tests with large data setups
3. ⏳ **Next:** Implement shared fixtures for policy tests
4. ✅ **Done:** Tag performance tests as `:slow` for conditional execution
5. ✅ **Done:** Nightly integration test suite for comprehensive coverage
---
## Slow Test Suite
Performance tests have been tagged with `@tag :slow` and separated into a dedicated test suite.
### Structure
- **Performance Tests:** Explicit tests that validate performance characteristics (e.g., N+1 query prevention, filter performance with large datasets)
- **Tagging:** All performance tests are tagged with `@tag :slow` or `@moduletag :slow`
- **Execution:** Standard test runs exclude slow tests, but they can be executed on demand
### Execution
**Fast Tests (Default):**
```bash
just test-fast
# or
mix test --exclude slow
# With specific files or options
just test-fast test/membership/member_test.exs
just test-fast --seed 123
```
**Performance Tests Only:**
```bash
just test-slow
# or
mix test --only slow
# With specific files or options
just test-slow test/mv_web/member_live/index_test.exs
just test-slow --seed 123
```
**All Tests:**
```bash
just test
# or
mix test
# With specific files or options
just test-all test/mv_web/
just test-all --max-failures 5
```
**Note:** All suite commands (`test-fast`, `test-slow`, `test-all`) support additional arguments. The suite semantics (tags) are always preserved - additional arguments are appended to the command.
### CI/CD Integration
- **Standard CI:** Runs `mix test --exclude slow` for faster feedback loops
- **Nightly Builds:** Separate pipeline (`nightly-tests`) runs daily at 2 AM and executes all performance tests
- **Manual Execution:**
- **Local:** Performance tests can be executed anytime with `just test-slow`
- **CI:** The nightly pipeline can be manually triggered via Drone CLI or Web UI:
```bash
drone build start <owner>/<repo> <branch> --event custom
```
### Further Details
See [test-slow-suite.md](test-slow-suite.md) for complete documentation of the Slow Test Suite.
---
## Monitoring Plan
### Success Criteria (Next 3 Months)
- ✅ Seeds tests execute in <20 seconds consistently
- ✅ No increase in seeds-related deployment failures
- ✅ No regression in authorization or membership fee bugs that would have been caught by removed tests
### What to Watch For
1. **Production Seeds Failures:**
- Monitor deployment logs for seeds errors
- If failures increase, consider restoring detailed tests
2. **Authorization Bugs After Seeds Changes:**
- If role/permission bugs appear after seeds modifications
- May indicate need for more seeds-specific role validation
3. **Developer Feedback:**
- If developers report missing test coverage
- Adjust based on real-world experience
---
## References
- **Detailed Documentation:** `docs/test-optimization-seeds.md`
- **Slow Test Suite:** `docs/test-slow-suite.md`
- **Test File:** `test/seeds_test.exs`
- **Original Analysis:** Internal benchmarking session (2026-01-28)
- **Related Guidelines:** `CODE_GUIDELINES.md` - Section 4 (Testing Standards)
---
## Changelog
### 2026-01-28: Initial Optimization
- Reduced seeds tests from 13 to 4
- Consolidated setup using per-test seeds execution
- Documented coverage mapping and risk assessment
- Measured time savings: 10-16 seconds per run
---
## Appendix: Full Test Output Example
```bash
$ mix test test/seeds_test.exs --slowest 10
Mv.SeedsTest [test/seeds_test.exs]
* test Seeds script is idempotent when run multiple times (5490.6ms)
* test Critical bootstrap invariants Mitglied system role exists (2630.2ms)
* test Seeds script runs successfully and creates basic data (2624.4ms)
* test Critical bootstrap invariants Admin user has Admin role (2619.0ms)
Finished in 13.6 seconds (0.00s async, 13.6s sync)
Top 10 slowest (13.3s), 98.1% of total time:
* test Seeds script is idempotent when run multiple times (5490.6ms)
* test Critical bootstrap invariants Mitglied system role exists (2630.2ms)
* test Seeds script runs successfully and creates basic data (2624.4ms)
* test Critical bootstrap invariants Admin user has Admin role (2619.0ms)
4 tests, 0 failures
```

543
docs/test-performance-optimization.md Normal file
View file

@ -0,0 +1,543 @@
# Test Performance Optimization
**Last Updated:** 2026-01-28
**Status:** ✅ Active optimization program
---
## Executive Summary
This document provides a comprehensive overview of test performance optimizations, risk assessments, and future opportunities. The test suite execution time has been reduced through systematic analysis and targeted optimizations.
### Current Performance Metrics
| Metric | Value |
|--------|-------|
| **Total Execution Time** (without `:slow` tests) | ~614 seconds (~10.2 minutes) |
| **Total Tests** | 1,368 tests (+ 25 doctests) |
| **Async Execution** | 317.7 seconds |
| **Sync Execution** | 296.2 seconds |
| **Slow Tests Excluded** | 9 tests (tagged with `@tag :slow`) |
| **Top 20 Slowest Tests** | 81.2 seconds (13.2% of total time) |
### Optimization Impact Summary
| Optimization | Tests Affected | Time Saved | Status |
|--------------|----------------|------------|--------|
| Seeds tests reduction | 13 → 4 tests | ~10-16s | ✅ Completed |
| Performance tests tagging | 9 tests | ~3-4s per run | ✅ Completed |
| Critical test query filtering | 1 test | ~8-10s | ✅ Completed |
| **Total Saved** | | **~21-30s** | |
---
## Completed Optimizations
### 1. Seeds Test Suite Optimization
**Date:** 2026-01-28
**Status:** ✅ Completed
#### What Changed
- **Reduced test count:** From 13 tests to 4 tests (69% reduction)
- **Reduced seeds executions:** From 8-10 times to 5 times per test run
- **Execution time:** From 24-30 seconds to 13-17 seconds
- **Time saved:** ~10-16 seconds per test run (40-50% faster)
#### Removed Tests (9 tests)
Tests were removed because their functionality is covered by domain-specific test suites:
1. `"at least one member has no membership fee type assigned"` → Covered by `membership_fees/*_test.exs`
2. `"each membership fee type has at least one member"` → Covered by `membership_fees/*_test.exs`
3. `"members with fee types have cycles with various statuses"` → Covered by `cycle_generator_test.exs`
4. `"creates all 5 authorization roles with correct permission sets"` → Covered by `authorization/*_test.exs`
5. `"all roles have valid permission_set_names"` → Covered by `authorization/permission_sets_test.exs`
6. `"does not change role of users who already have a role"` → Merged into idempotency test
7. `"role creation is idempotent"` (detailed) → Merged into general idempotency test
#### Retained Tests (4 tests)
Critical deployment requirements are still covered:
1. ✅ **Smoke Test:** Seeds run successfully and create basic data
2. ✅ **Idempotency Test:** Seeds can be run multiple times without duplicating data
3. ✅ **Admin Bootstrap:** Admin user exists with Admin role (critical for initial access)
4. ✅ **System Role Bootstrap:** Mitglied system role exists (critical for user registration)
#### Risk Assessment
| Removed Test Category | Alternative Coverage | Risk Level |
|----------------------|---------------------|------------|
| Member/fee type distribution | `membership_fees/*_test.exs` | ⚠️ Low |
| Cycle status variations | `cycle_generator_test.exs` | ⚠️ Low |
| Detailed role configs | `authorization/*_test.exs` | ⚠️ Very Low |
| Permission set validation | `permission_sets_test.exs` | ⚠️ Very Low |
**Overall Risk:** ⚠️ **Low** - All removed tests have equivalent or better coverage in domain-specific test suites.
---
### 2. Performance Test Suite Separation
**Date:** 2026-01-28
**Status:** ✅ Completed
#### What Changed
Performance tests that explicitly validate performance characteristics are now tagged with `@tag :slow` or `@moduletag :slow` and excluded from standard test runs.
#### Identified Performance Tests (9 tests)
1. **Member LiveView - Boolean Filter Performance** (`test/mv_web/member_live/index_test.exs`)
- Test: `"boolean filter performance with 150 members"`
- Duration: ~3.8 seconds
- Creates 150 members to test filter performance
2. **Group LiveView - Index Performance** (`test/mv_web/live/group_live/index_test.exs`)
- 2 tests validating efficient page loading and member count calculation
3. **Group LiveView - Show Performance** (`test/mv_web/live/group_live/show_test.exs`)
- 2 tests validating efficient member list loading and slug lookup
4. **Member LiveView - Membership Fee Status Performance** (`test/mv_web/member_live/index_membership_fee_status_test.exs`)
- 1 test validating efficient cycle loading without N+1 queries
5. **Group Integration - Query Performance** (`test/membership/group_integration_test.exs`)
- 1 test validating N+1 query prevention for group-member relationships
#### Execution Commands
**Fast Tests (Default):**
```bash
just test-fast
# or
mix test --exclude slow
```
**Performance Tests Only:**
```bash
just test-slow
# or
mix test --only slow
```
**All Tests:**
```bash
just test
# or
mix test
```
#### CI/CD Integration
- **Standard CI:** Runs `mix test --exclude slow` for faster feedback loops
- **Nightly Builds:** Separate pipeline (`nightly-tests`) runs daily at 2 AM and executes all performance tests
- **Manual Execution:** Can be triggered via Drone CLI or Web UI
#### Risk Assessment
**Risk Level:** ✅ **Very Low**
- Performance tests are still executed, just separately
- Standard test runs are faster, improving developer feedback
- Nightly builds ensure comprehensive coverage
- No functionality is lost, only execution timing changed
---
### 3. Critical Test Optimization
**Date:** 2026-01-28
**Status:** ✅ Completed
#### Problem Identified
The test `test respects show_in_overview config` was the slowest test in the suite:
- **Isolated execution:** 4.8 seconds
- **In full test run:** 14.7 seconds
- **Difference:** 9.9 seconds (test isolation issue)
#### Root Cause
The test loaded **all members** from the database, not just the 2 members from the test setup. In full test runs, many members from other tests were present in the database, significantly slowing down the query.
#### Solution Implemented
**Query Filtering:** Added search query parameter to filter to only the expected member.
**Code Change:**
```elixir
# Before:
{:ok, _view, html} = live(conn, "/members")
# After:
{:ok, _view, html} = live(conn, "/members?query=Alice")
```
#### Results
| Execution | Before | After | Improvement |
|-----------|--------|-------|-------------|
| **Isolated** | 4.8s | 1.1s | **-77%** (3.7s saved) |
| **In Module** | 4.2s | 0.4s | **-90%** (3.8s saved) |
| **Expected in Full Run** | 14.7s | ~4-6s | **-65% to -73%** (8-10s saved) |
#### Risk Assessment
**Risk Level:** ✅ **Very Low**
- Test functionality unchanged - only loads expected data
- All assertions still pass
- Test is now faster and more isolated
- No impact on test coverage
---
## Current Performance Analysis
### Top 20 Slowest Tests (without `:slow`)
| Rank | Test | File | Time | % of Top 20 |
|------|------|------|------|-------------|
| 1 | `test respects show_in_overview config` | `index_member_fields_display_test.exs` | **~4-6s** (optimized) | ~6-7% |
| 2 | `test Seeds script is idempotent when run multiple times` | `seeds_test.exs` | 5.0s | 6.2% |
| 3 | `test sorting functionality initially sorts by email ascending` | `user_live/index_test.exs` | 4.5s | 5.5% |
| 4 | `test Seeds script runs successfully and creates basic data` | `seeds_test.exs` | 4.3s | 5.3% |
| 5-20 | Various User LiveView and Policy tests | Multiple files | 2.6-4.2s each | 3-5% each |
**Total Top 20:** ~81 seconds (13.2% of total time)
### Performance Hotspots Identified
#### 1. Seeds Tests (~16.2s for 4 tests)
**Status:** ✅ Optimized (reduced from 13 tests)
**Remaining Optimization Potential:** 3-5 seconds
**Opportunities:**
- Settings update could potentially be moved to `setup_all` (if sandbox allows)
- Seeds execution could be further optimized (less data in test mode)
- Idempotency test could be optimized (only 1x seeds instead of 2x)
#### 2. User LiveView Tests (~35.5s for 10 tests)
**Status:** ⏳ Identified for optimization
**Optimization Potential:** 15-20 seconds
**Files:**
- `test/mv_web/user_live/index_test.exs` (3 tests, ~10.2s)
- `test/mv_web/user_live/form_test.exs` (4 tests, ~15.0s)
- `test/mv_web/user_live/show_test.exs` (3 tests, ~10.3s)
**Patterns:**
- Many tests create user/member data
- LiveView mounts are expensive
- Form submissions with validations are slow
**Recommended Actions:**
- Move shared fixtures to `setup_all`
- Reduce test data volume (3-5 users instead of 10+)
- Optimize setup patterns for recurring patterns
#### 3. Policy Tests (~8.7s for 3 tests)
**Status:** ⏳ Identified for optimization
**Optimization Potential:** 5-8 seconds
**Files:**
- `test/mv/membership/member_policies_test.exs` (2 tests, ~6.1s)
- `test/mv/accounts/user_policies_test.exs` (1 test, ~2.6s)
**Pattern:**
- Each test creates new roles/users/members
- Roles are identical across tests
**Recommended Actions:**
- Create roles in `setup_all` (shared across tests)
- Reuse common fixtures
- Maintain test isolation while optimizing setup
---
## Future Optimization Opportunities
### Priority 1: User LiveView Tests Optimization
**Estimated Savings:** 15-20 seconds
**Actions:**
1. Move shared fixtures to `setup_all` where possible
2. Reduce test data volume (use 3-5 users instead of 10+)
3. Analyze and optimize recurring setup patterns
4. Consider mocking expensive operations (if appropriate)
**Risk Assessment:** ⚠️ **Low**
- Requires careful testing to ensure isolation is maintained
- Should verify that shared fixtures don't cause test interdependencies
### Priority 2: Policy Tests Optimization
**Estimated Savings:** 5-8 seconds
**Actions:**
1. Create roles in `setup_all` (they're identical across tests)
2. Reuse common fixtures (users, members)
3. Maintain test isolation while optimizing setup
**Risk Assessment:** ⚠️ **Low**
- Roles are read-only in tests, safe to share
- Need to ensure user/member fixtures don't interfere with each other
### Priority 3: Seeds Tests Further Optimization
**Estimated Savings:** 3-5 seconds
**Actions:**
1. Investigate if settings update can be moved to `setup_all`
2. Introduce seeds mode for tests (less data in test mode)
3. Optimize idempotency test (only 1x seeds instead of 2x)
**Risk Assessment:** ⚠️ **Low to Medium**
- Sandbox limitations may prevent `setup_all` usage
- Seeds mode would require careful implementation
- Idempotency test optimization needs to maintain test validity
### Priority 4: Additional Test Isolation Improvements
**Estimated Savings:** Variable (depends on specific tests)
**Actions:**
1. Review tests that load all records (similar to the critical test fix)
2. Add query filters where appropriate
3. Ensure proper test isolation in async tests
**Risk Assessment:** ⚠️ **Very Low**
- Similar to the critical test optimization (proven approach)
- Improves test isolation and reliability
---
## Estimated Total Optimization Potential
| Priority | Optimization | Estimated Savings |
|----------|-------------|-------------------|
| 1 | User LiveView Tests | 15-20s |
| 2 | Policy Tests | 5-8s |
| 3 | Seeds Tests Further | 3-5s |
| 4 | Additional Isolation | Variable |
| **Total Potential** | | **23-33 seconds** |
**Projected Final Time:** From ~614 seconds to **~580-590 seconds** (~9.5-10 minutes)
---
## Risk Assessment Summary
### Overall Risk Level: ⚠️ **Low**
All optimizations maintain test coverage while improving performance:
| Optimization | Risk Level | Mitigation |
|-------------|------------|------------|
| Seeds tests reduction | ⚠️ Low | Coverage mapped to domain tests |
| Performance tests tagging | ✅ Very Low | Tests still executed, just separately |
| Critical test optimization | ✅ Very Low | Functionality unchanged, better isolation |
| Future optimizations | ⚠️ Low | Careful implementation with verification |
### Monitoring Plan
#### Success Criteria
- ✅ Seeds tests execute in <20 seconds consistently
- ✅ No increase in seeds-related deployment failures
- ✅ No regression in authorization or membership fee bugs
- ⏳ Top 20 slowest tests: < 60 seconds (currently 81.2s)
- ⏳ Total execution time (without `:slow`): < 10 minutes (currently 10.2 min)
#### What to Watch For
1. **Production Seeds Failures:**
- Monitor deployment logs for seeds errors
- If failures increase, consider restoring detailed tests
2. **Authorization Bugs After Seeds Changes:**
- If role/permission bugs appear after seeds modifications
- May indicate need for more seeds-specific role validation
3. **Test Performance Regression:**
- Monitor test execution times in CI
- Alert if times increase significantly
4. **Developer Feedback:**
- If developers report missing test coverage
- Adjust based on real-world experience
---
## Benchmarking and Analysis
### How to Benchmark Tests
**ExUnit Built-in Benchmarking:**
The test suite is configured to show the slowest tests automatically:
```elixir
# test/test_helper.exs
ExUnit.start(
slowest: 10 # Shows 10 slowest tests at the end of test run
)
```
**Run Benchmark Analysis:**
```bash
# Show slowest tests
mix test --slowest 20
# Exclude slow tests for faster feedback
mix test --exclude slow --slowest 20
# Run only slow tests
mix test --only slow --slowest 10
# Benchmark specific test file
mix test test/mv_web/member_live/index_member_fields_display_test.exs --slowest 5
```
### Benchmarking Best Practices
1. **Run benchmarks regularly** (e.g., monthly) to catch performance regressions
2. **Compare isolated vs. full runs** to identify test isolation issues
3. **Monitor CI execution times** to track trends over time
4. **Document significant changes** in test performance
---
## Test Suite Structure
### Test Execution Modes
**Fast Tests (Default):**
- Excludes performance tests (`@tag :slow`)
- Used for standard development workflow
- Command: `mix test --exclude slow` or `just test-fast`
**Performance Tests:**
- Explicitly tagged performance tests
- Run separately or in nightly builds
- Command: `mix test --only slow` or `just test-slow`
**All Tests:**
- Includes both fast and slow tests
- Used for comprehensive validation
- Command: `mix test` or `just test`
### Test Organization
Tests are organized to mirror the `lib/` directory structure:
```
test/
├── accounts/ # Accounts domain tests
├── membership/ # Membership domain tests
├── membership_fees/ # Membership fees domain tests
├── mv/ # Core application tests
│ ├── accounts/ # User-related tests
│ ├── membership/ # Member-related tests
│ └── authorization/ # Authorization tests
├── mv_web/ # Web layer tests
│ ├── controllers/ # Controller tests
│ ├── live/ # LiveView tests
│ └── components/ # Component tests
└── support/ # Test helpers
├── conn_case.ex # Controller test setup
└── data_case.ex # Database test setup
```
---
## Best Practices for Test Performance
### When Writing New Tests
1. **Use `async: true`** when possible (for parallel execution)
2. **Filter queries** to only load necessary data
3. **Share fixtures** in `setup_all` when appropriate
4. **Tag performance tests** with `@tag :slow` if they use large datasets
5. **Keep test data minimal** - only create what's needed for the test
### When Optimizing Existing Tests
1. **Measure first** - Use `mix test --slowest` to identify bottlenecks
2. **Compare isolated vs. full runs** - Identify test isolation issues
3. **Optimize setup** - Move shared data to `setup_all` where possible
4. **Filter queries** - Only load data needed for the test
5. **Verify coverage** - Ensure optimizations don't reduce test coverage
### Performance Test Guidelines
**Tag as `:slow` when:**
- Explicitly testing performance characteristics
- Using large datasets (50+ records)
- Testing scalability or query optimization
- Validating N+1 query prevention
**Do NOT tag as `:slow` when:**
- Test is slow due to inefficient setup (fix the setup instead)
- Test is slow due to bugs (fix the bug instead)
- It's an integration test (use `@tag :integration` instead)
---
## Changelog
### 2026-01-28: Initial Optimization Phase
**Completed:**
- ✅ Reduced seeds tests from 13 to 4 tests
- ✅ Tagged 9 performance tests with `@tag :slow`
- ✅ Optimized critical test with query filtering
- ✅ Created slow test suite infrastructure
- ✅ Updated CI/CD to exclude slow tests from standard runs
- ✅ Added nightly CI pipeline for slow tests
**Time Saved:** ~21-30 seconds per test run
**Next Steps:**
- ⏳ Optimize User LiveView tests (Priority 1)
- ⏳ Optimize Policy tests (Priority 2)
- ⏳ Further optimize Seeds tests (Priority 3)
---
## References
- **Testing Standards:** `CODE_GUIDELINES.md` - Section 4 (Testing Standards)
- **CI/CD Configuration:** `.drone.yml`
- **Test Helper:** `test/test_helper.exs`
- **Justfile Commands:** `Justfile` (test-fast, test-slow, test-all)
---
## Questions & Answers
**Q: What if seeds create wrong data and break the system?**
A: The smoke test will fail if seeds raise errors. Domain tests ensure business logic is correct regardless of seeds content.
**Q: What if we add a new critical bootstrap requirement?**
A: Add a new test to the "Critical bootstrap invariants" section in `test/seeds_test.exs`.
**Q: How do we know the removed tests aren't needed?**
A: Monitor for 2-3 months. If no seeds-related bugs appear that would have been caught by removed tests, they were redundant.
**Q: Should we restore the tests for important releases?**
A: Consider running the full test suite (including slow tests) before major releases. Daily development uses the optimized suite.
**Q: How do I add a new performance test?**
A: Tag it with `@tag :slow` or `@moduletag :slow`. See "Performance Test Guidelines" section above.
**Q: Can I run slow tests locally?**
A: Yes, use `just test-slow` or `mix test --only slow`. They're excluded from standard runs for faster feedback.

300
docs/test-slow-suite.md
View file

@ -1,300 +0,0 @@
# Slow Test Suite Documentation
**Date:** 2026-01-28
**Status:** ✅ Active
---
## Overview
The Slow Test Suite contains performance tests that explicitly validate performance characteristics of the application. These tests are intentionally slower because they use larger datasets or test performance-critical paths (e.g., N+1 query prevention, filter performance with many records).
These tests are marked with `@tag :slow` or `@moduletag :slow` and are excluded from standard test runs to improve developer feedback loops while maintaining comprehensive coverage.
---
## Purpose
Performance tests serve to:
1. **Validate Performance Characteristics:** Ensure queries and operations perform within acceptable time limits
2. **Prevent Regressions:** Catch performance regressions before they reach production
3. **Test Scalability:** Verify that the application handles larger datasets efficiently
4. **N+1 Query Prevention:** Ensure proper preloading and query optimization
---
## Identified Performance Tests
### 1. Member LiveView - Boolean Filter Performance
**File:** `test/mv_web/member_live/index_test.exs`
**Test:** `"boolean filter performance with 150 members"`
**Duration:** ~3.8 seconds
**Purpose:** Validates that boolean custom field filtering performs efficiently with 150 members
**What it tests:**
- Creates 150 members (75 with `true`, 75 with `false` for a boolean custom field)
- Tests filter performance (< 1 second requirement)
- Verifies correct filtering behavior
### 2. Group LiveView - Index Performance
**File:** `test/mv_web/live/group_live/index_test.exs`
**Describe Block:** `"performance"`
**Tests:** 2 tests
**Purpose:** Validates efficient page loading and member count calculation
**What it tests:**
- Page loads efficiently with many groups (no N+1 queries)
- Member count calculation is efficient
### 3. Group LiveView - Show Performance
**File:** `test/mv_web/live/group_live/show_test.exs`
**Describe Block:** `"performance"`
**Tests:** 2 tests
**Purpose:** Validates efficient member list loading and slug lookup
**What it tests:**
- Member list is loaded efficiently (no N+1 queries)
- Slug lookup uses unique_slug index efficiently
### 4. Member LiveView - Membership Fee Status Performance
**File:** `test/mv_web/member_live/index_membership_fee_status_test.exs`
**Describe Block:** `"performance"`
**Tests:** 1 test
**Purpose:** Validates efficient cycle loading without N+1 queries
**What it tests:**
- Cycles are loaded efficiently without N+1 queries
- Multiple members with cycles render without performance issues
### 5. Group Integration - Query Performance
**File:** `test/membership/group_integration_test.exs`
**Describe Block:** `"Query Performance"`
**Tests:** 1 test
**Purpose:** Validates N+1 query prevention for group-member relationships
**What it tests:**
- Preloading groups with members avoids N+1 queries
- Query optimization for many-to-many relationships
---
## Running Slow Tests
### Local Development
**Run only fast tests (default):**
```bash
just test-fast
# or
mix test --exclude slow
# With specific files or options
just test-fast test/membership/member_test.exs
just test-fast --seed 123
```
**Run only performance tests:**
```bash
just test-slow
# or
mix test --only slow
# With specific files or options
just test-slow test/mv_web/member_live/index_test.exs
just test-slow --seed 123
```
**Run all tests (fast + slow):**
```bash
just test
# or
mix test
# With specific files or options
just test-all test/mv_web/
just test-all --max-failures 5
```
**Note:** All suite commands (`test-fast`, `test-slow`, `test-all`) support additional arguments. The suite semantics (tags) are always preserved - additional arguments are appended to the command.
### CI/CD
**Standard CI Pipeline:**
- Runs `mix test --exclude slow` for faster feedback
- Executes on every push to any branch
**Nightly Pipeline:**
- Runs `mix test --only slow` for comprehensive performance coverage
- Executes daily at 2 AM via cron trigger
- Pipeline name: `nightly-tests`
**Manual Execution:**
The nightly pipeline can be manually triggered using:
**Drone CLI:**
```bash
drone build start <owner>/<repo> <branch> --event custom
```
**Drone Web UI:**
- Navigate to the repository in Drone
- Go to the `nightly-tests` pipeline
- Click "Run" or "Restart" and select event type `custom`
**Example:**
```bash
# Trigger nightly-tests pipeline manually
drone build start local-it/mitgliederverwaltung main --event custom
```
---
## Best Practices for New Performance Tests
### When to Tag as `:slow`
Tag tests as `:slow` when they:
1. **Explicitly test performance:** Tests that measure execution time or validate performance characteristics
2. **Use large datasets:** Tests that create many records (e.g., 50+ members, 100+ records)
3. **Test scalability:** Tests that verify the application handles larger workloads
4. **Validate query optimization:** Tests that ensure N+1 queries are prevented
### When NOT to Tag as `:slow`
Do NOT tag tests as `:slow` if they are:
1. **Simply slow by accident:** Tests that are slow due to inefficient setup, not intentional performance testing
2. **Slow due to bugs:** Tests that are slow because of actual performance bugs (fix the bug instead)
3. **Integration tests:** Integration tests should be tagged separately if needed (`@tag :integration`)
### Tagging Guidelines
**For single tests:**
```elixir
@tag :slow
test "boolean filter performance with 150 members" do
# test implementation
end
```
**For describe blocks (all tests in block):**
```elixir
@moduletag :slow
describe "performance" do
test "page loads efficiently" do
# test implementation
end
test "member count is efficient" do
# test implementation
end
end
```
---
## Performance Test Structure
### Recommended Structure
```elixir
defmodule MvWeb.SomeLiveViewTest do
use MvWeb.ConnCase, async: true
# Regular tests here (not tagged)
@moduletag :slow
describe "performance" do
test "loads efficiently without N+1 queries" do
# Create test data
# Measure/validate performance
# Assert correct behavior
end
test "handles large datasets efficiently" do
# Create large dataset
# Measure performance
# Assert performance requirements
end
end
end
```
### Performance Assertions
Performance tests should include explicit performance assertions:
```elixir
# Example: Time-based assertion
start_time = System.monotonic_time(:millisecond)
# ... perform operation ...
end_time = System.monotonic_time(:millisecond)
duration = end_time - start_time
assert duration < 1000, "Operation took #{duration}ms, expected < 1000ms"
```
```elixir
# Example: Query count assertion (using Ecto query logging)
# Verify no N+1 queries by checking query count
```
---
## Monitoring and Maintenance
### Regular Review
- **Quarterly Review:** Review slow tests quarterly to ensure they're still relevant
- **Performance Baselines:** Update performance assertions if legitimate performance improvements occur
- **Test Cleanup:** Remove or optimize tests that become redundant
### Success Metrics
- ✅ Performance tests catch regressions before production
- ✅ Standard test runs complete in < 3 minutes
- ✅ Nightly builds complete successfully
- ✅ No false positives in performance tests
### Troubleshooting
**If a performance test fails:**
1. **Check if it's a real regression:** Compare with previous runs
2. **Check CI environment:** Ensure CI has adequate resources
3. **Review test data:** Ensure test data setup is correct
4. **Check for flakiness:** Run test multiple times to verify consistency
**If a performance test is too slow:**
1. **Review test implementation:** Look for inefficiencies in test setup
2. **Consider reducing dataset size:** If still representative
3. **Split into smaller tests:** If testing multiple concerns
---
## Related Documentation
- **Test Optimization Summary:** `docs/test-optimization-summary.md`
- **Seeds Test Optimization:** `docs/test-optimization-seeds.md`
- **Testing Standards:** `CODE_GUIDELINES.md` - Section 4 (Testing Standards)
- **CI/CD Configuration:** `.drone.yml`
---
## Changelog
### 2026-01-28: Initial Setup
- Marked 5 performance test suites with `@tag :slow` or `@moduletag :slow`
- Added `test-fast`, `test-slow`, and `test-all` commands to Justfile
- Updated CI to exclude slow tests from standard runs
- Added nightly CI pipeline for slow tests
- Created this documentation