docs: consolidate test performance docs
This commit is contained in:
parent
9b314a9806
commit
709cf010c6
GPG key ID:
40E7A58C4AA1EDB2
3 changed files with 183 additions and 1411 deletions
|
|
@ -373,30 +373,190 @@ After implementing the full test suite via promotion, the remaining slowest test
|
|||
|
||||
### Priority 1: User LiveView Tests Optimization
|
||||
|
||||
**Estimated Savings:** 15-20 seconds
|
||||
**Estimated Savings:** 14-22 seconds
|
||||
**Status:** 📋 Analysis Complete - Ready for Implementation
|
||||
|
||||
**Actions:**
|
||||
1. Move shared fixtures to `setup_all` where possible
|
||||
2. Reduce test data volume (use 3-5 users instead of 10+)
|
||||
3. Analyze and optimize recurring setup patterns
|
||||
4. Consider mocking expensive operations (if appropriate)
|
||||
#### Analysis Summary
|
||||
|
||||
Analysis of User LiveView tests identified significant optimization opportunities:
|
||||
- **Framework functionality over-testing:** ~30 tests test Phoenix/Ash/Gettext core features
|
||||
- **Redundant test data creation:** Each test creates users/members independently
|
||||
- **Missing shared fixtures:** No `setup_all` usage for common data
|
||||
|
||||
#### Current Performance
|
||||
|
||||
**Top 20 Slowest Tests (User LiveView):**
|
||||
- `index_test.exs`: ~10.2s for 3 tests in Top 20
|
||||
- `form_test.exs`: ~15.0s for 4 tests in Top 20
|
||||
- `show_test.exs`: ~10.3s for 3 tests in Top 20
|
||||
- **Total:** ~35.5 seconds for User LiveView tests
|
||||
|
||||
#### Optimization Opportunities
|
||||
|
||||
**1. Remove Framework Functionality Tests (~30 tests, 8-12s saved)**
|
||||
- Remove translation tests (Gettext framework)
|
||||
- Remove navigation tests (Phoenix LiveView framework)
|
||||
- Remove validation tests (Ash framework)
|
||||
- Remove basic HTML rendering tests (consolidate into smoke test)
|
||||
- Remove password storage tests (AshAuthentication framework)
|
||||
|
||||
**2. Implement Shared Fixtures (3-5s saved)**
|
||||
- Use `setup_all` for common test data in `index_test.exs` and `show_test.exs`
|
||||
- Share users for sorting/checkbox tests
|
||||
- Share common users/members across tests
|
||||
- **Note:** `form_test.exs` uses `async: false`, preventing `setup_all` usage
|
||||
|
||||
**3. Consolidate Redundant Tests (~10 tests → 3-4 tests, 2-3s saved)**
|
||||
- Merge basic display tests into smoke test
|
||||
- Merge navigation tests into integration test
|
||||
- Reduce sorting tests to 1 integration test
|
||||
|
||||
**4. Optimize Test Data Volume (1-2s saved)**
|
||||
- Use minimum required data (2 users for sorting, 2 for checkboxes)
|
||||
- Share data across tests via `setup_all`
|
||||
|
||||
#### Tests to Keep (Business Logic)
|
||||
|
||||
**Index Tests:**
|
||||
- `initially sorts by email ascending` - Tests default sort
|
||||
- `can sort email descending by clicking sort button` - Tests sort functionality
|
||||
- `select all automatically checks when all individual users are selected` - Business logic
|
||||
- `does not show system actor user in list` - Business rule
|
||||
- `displays linked member name in user list` - Business logic
|
||||
- Edge case tests
|
||||
|
||||
**Form Tests:**
|
||||
- `creates user without password` - Business logic
|
||||
- `creates user with password when enabled` - Business logic
|
||||
- `admin sets new password for user` - Business logic
|
||||
- `selecting member and saving links member to user` - Business logic
|
||||
- Member linking/unlinking workflow tests
|
||||
|
||||
**Show Tests:**
|
||||
- `displays password authentication status` - Business logic
|
||||
- `displays linked member when present` - Business logic
|
||||
- `redirects to user list when viewing system actor user` - Business rule
|
||||
|
||||
#### Implementation Plan
|
||||
|
||||
**Phase 1: Remove Framework Tests (1-2 hours, ⚠️ Very Low Risk)**
|
||||
- Remove translation, navigation, validation, and basic HTML rendering tests
|
||||
- Consolidate remaining display tests into smoke test
|
||||
|
||||
**Phase 2: Implement Shared Fixtures (2-3 hours, ⚠️ Low Risk)**
|
||||
- Add `setup_all` to `index_test.exs` and `show_test.exs`
|
||||
- Update tests to use shared fixtures
|
||||
- Verify test isolation maintained
|
||||
|
||||
**Phase 3: Consolidate Tests (1-2 hours, ⚠️ Very Low Risk)**
|
||||
- Merge basic display tests into smoke test
|
||||
- Merge navigation tests into integration test
|
||||
- Reduce sorting tests to 1 integration test
|
||||
|
||||
**Risk Assessment:** ⚠️ **Low**
|
||||
- Requires careful testing to ensure isolation is maintained
|
||||
- Should verify that shared fixtures don't cause test interdependencies
|
||||
- Framework functionality is tested by framework maintainers
|
||||
- Business logic tests remain intact
|
||||
- Shared fixtures maintain test isolation
|
||||
- Consolidation preserves coverage
|
||||
|
||||
### Priority 2: Policy Tests Optimization
|
||||
|
||||
**Estimated Savings:** 5-8 seconds
|
||||
**Estimated Savings:** 5.5-9 seconds
|
||||
**Status:** 📋 Analysis Complete - Ready for Decision
|
||||
|
||||
**Actions:**
|
||||
1. Create roles in `setup_all` (they're identical across tests)
|
||||
2. Reuse common fixtures (users, members)
|
||||
3. Maintain test isolation while optimizing setup
|
||||
#### Analysis Summary
|
||||
|
||||
Analysis of policy tests identified significant optimization opportunities:
|
||||
- **Redundant fixture creation:** Roles and users created repeatedly across tests
|
||||
- **Framework functionality over-testing:** Many tests verify Ash policy framework behavior
|
||||
- **Test duplication:** Similar tests across different permission sets
|
||||
|
||||
#### Current Performance
|
||||
|
||||
**Policy Test Files Performance:**
|
||||
- `member_policies_test.exs`: 24 tests, ~66s (top 20)
|
||||
- `user_policies_test.exs`: 30 tests, ~66s (top 20)
|
||||
- `custom_field_value_policies_test.exs`: 20 tests, ~66s (top 20)
|
||||
- **Total:** 74 tests, ~152s total
|
||||
|
||||
**Top 20 Slowest Policy Tests:** ~66 seconds
|
||||
|
||||
#### Framework vs. Business Logic Analysis
|
||||
|
||||
**Framework Functionality (Should NOT Test):**
|
||||
- Policy evaluation (how Ash evaluates policies)
|
||||
- Permission lookup (how Ash looks up permissions)
|
||||
- Scope filtering (how Ash applies scope filters)
|
||||
- Auto-filter behavior (how Ash auto-filters queries)
|
||||
- Forbidden vs NotFound (how Ash returns errors)
|
||||
|
||||
**Business Logic (Should Test):**
|
||||
- Permission set definitions (what permissions each role has)
|
||||
- Scope definitions (what scopes each permission set uses)
|
||||
- Special cases (custom business rules)
|
||||
- Permission set behavior (how our permission sets differ)
|
||||
|
||||
#### Optimization Opportunities
|
||||
|
||||
**1. Remove Framework Functionality Tests (~22-34 tests, 3-4s saved)**
|
||||
- Remove "cannot" tests that verify error types (Forbidden, NotFound)
|
||||
- Remove tests that verify auto-filter behavior (framework)
|
||||
- Remove tests that verify permission evaluation (framework)
|
||||
- **Risk:** ⚠️ Very Low - Framework functionality is tested by Ash maintainers
|
||||
|
||||
**2. Consolidate Redundant Tests (~6-8 tests → 2-3 tests, 1-2s saved)**
|
||||
- Merge similar tests across permission sets
|
||||
- Create integration tests that cover multiple permission sets
|
||||
- **Risk:** ⚠️ Low - Same coverage, fewer tests
|
||||
|
||||
**3. Share Admin User Across Describe Blocks (1-2s saved)**
|
||||
- Create admin user once in module-level `setup`
|
||||
- Reuse admin user in helper functions
|
||||
- **Note:** `async: false` prevents `setup_all`, but module-level `setup` works
|
||||
- **Risk:** ⚠️ Low - Admin user is read-only in tests, safe to share
|
||||
|
||||
**4. Reduce Test Data Volume (0.5-1s saved)**
|
||||
- Use minimum required data
|
||||
- Share fixtures where possible
|
||||
- **Risk:** ⚠️ Very Low - Still tests same functionality
|
||||
|
||||
#### Test Classification Summary
|
||||
|
||||
**Tests to Remove (Framework):**
|
||||
- `member_policies_test.exs`: ~10 tests (cannot create/destroy/update, auto-filter tests)
|
||||
- `user_policies_test.exs`: ~16 tests (cannot read/update/create/destroy, auto-filter tests)
|
||||
- `custom_field_value_policies_test.exs`: ~8 tests (similar "cannot" tests)
|
||||
|
||||
**Tests to Consolidate (Redundant):**
|
||||
- `user_policies_test.exs`: 6 tests → 2 tests (can read/update own user record)
|
||||
|
||||
**Tests to Keep (Business Logic):**
|
||||
- All "can" tests that verify permission set behavior
|
||||
- Special case tests (e.g., "user can always READ linked member")
|
||||
- AshAuthentication bypass tests (our integration)
|
||||
|
||||
#### Implementation Plan
|
||||
|
||||
**Phase 1: Remove Framework Tests (1-2 hours, ⚠️ Very Low Risk)**
|
||||
- Identify all "cannot" tests that verify error types
|
||||
- Remove tests that verify Ash auto-filter behavior
|
||||
- Remove tests that verify permission evaluation (framework)
|
||||
|
||||
**Phase 2: Consolidate Redundant Tests (1-2 hours, ⚠️ Low Risk)**
|
||||
- Identify similar tests across permission sets
|
||||
- Create integration tests that cover multiple permission sets
|
||||
- Remove redundant individual tests
|
||||
|
||||
**Phase 3: Share Admin User (1-2 hours, ⚠️ Low Risk)**
|
||||
- Add module-level `setup` to create admin user
|
||||
- Update helper functions to accept admin user parameter
|
||||
- Update all `setup` blocks to use shared admin user
|
||||
|
||||
**Risk Assessment:** ⚠️ **Low**
|
||||
- Roles are read-only in tests, safe to share
|
||||
- Need to ensure user/member fixtures don't interfere with each other
|
||||
- Framework functionality is tested by Ash maintainers
|
||||
- Business logic tests remain intact
|
||||
- Admin user sharing maintains test isolation (read-only)
|
||||
- Consolidation preserves coverage
|
||||
|
||||
### Priority 3: Seeds Tests Further Optimization
|
||||
|
||||
|
|
@ -431,13 +591,17 @@ After implementing the full test suite via promotion, the remaining slowest test
|
|||
|
||||
| Priority | Optimization | Estimated Savings |
|
||||
|----------|-------------|-------------------|
|
||||
| 1 | User LiveView Tests | 15-20s |
|
||||
| 2 | Policy Tests | 5-8s |
|
||||
| 1 | User LiveView Tests | 14-22s |
|
||||
| 2 | Policy Tests | 5.5-9s |
|
||||
| 3 | Seeds Tests Further | 3-5s |
|
||||
| 4 | Additional Isolation | Variable |
|
||||
| **Total Potential** | | **23-33 seconds** |
|
||||
| **Total Potential** | | **22.5-36 seconds** |
|
||||
|
||||
**Projected Final Time:** From ~368 seconds (fast suite) to **~340-350 seconds** (~5.7-5.8 minutes) with remaining optimizations
|
||||
**Projected Final Time:** From ~368 seconds (fast suite) to **~332-345 seconds** (~5.5-5.8 minutes) with remaining optimizations
|
||||
|
||||
**Note:** Detailed analysis documents available:
|
||||
- User LiveView Tests: See "Priority 1: User LiveView Tests Optimization" section above
|
||||
- Policy Tests: See "Priority 2: Policy Tests Optimization" section above
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue