|
|
|
|
@ -13,12 +13,12 @@ This document provides a comprehensive overview of test performance optimization
|
|
|
|
|
|
|
|
|
|
| Metric | Value |
|
|
|
|
|
|--------|-------|
|
|
|
|
|
| **Total Execution Time** (without `:slow` tests) | ~614 seconds (~10.2 minutes) |
|
|
|
|
|
| **Total Tests** | 1,368 tests (+ 25 doctests) |
|
|
|
|
|
| **Async Execution** | 317.7 seconds |
|
|
|
|
|
| **Sync Execution** | 296.2 seconds |
|
|
|
|
|
| **Slow Tests Excluded** | 9 tests (tagged with `@tag :slow`) |
|
|
|
|
|
| **Top 20 Slowest Tests** | 81.2 seconds (13.2% of total time) |
|
|
|
|
|
| **Total Execution Time** (without `:slow` tests) | ~368 seconds (~6.1 minutes) |
|
|
|
|
|
| **Total Tests** | 1,336 tests (+ 25 doctests) |
|
|
|
|
|
| **Async Execution** | 163.5 seconds |
|
|
|
|
|
| **Sync Execution** | 281.5 seconds |
|
|
|
|
|
| **Slow Tests Excluded** | 25 tests (tagged with `@tag :slow`) |
|
|
|
|
|
| **Top 50 Slowest Tests** | 121.9 seconds (27.4% of total time) |
|
|
|
|
|
|
|
|
|
|
### Optimization Impact Summary
|
|
|
|
|
|
|
|
|
|
@ -27,7 +27,8 @@ This document provides a comprehensive overview of test performance optimization
|
|
|
|
|
| Seeds tests reduction | 13 → 4 tests | ~10-16s | ✅ Completed |
|
|
|
|
|
| Performance tests tagging | 9 tests | ~3-4s per run | ✅ Completed |
|
|
|
|
|
| Critical test query filtering | 1 test | ~8-10s | ✅ Completed |
|
|
|
|
|
| **Total Saved** | | **~21-30s** | |
|
|
|
|
|
| Nightly suite tagging | 25 tests | ~77s per run | ✅ Completed |
|
|
|
|
|
| **Total Saved** | | **~98-107s** | |
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
@ -79,33 +80,71 @@ Critical deployment requirements are still covered:
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### 2. Performance Test Suite Separation
|
|
|
|
|
### 2. Nightly Suite Implementation (`@tag :slow`)
|
|
|
|
|
|
|
|
|
|
**Date:** 2026-01-28
|
|
|
|
|
**Status:** ✅ Completed
|
|
|
|
|
|
|
|
|
|
#### What Changed
|
|
|
|
|
|
|
|
|
|
Performance tests that explicitly validate performance characteristics are now tagged with `@tag :slow` or `@moduletag :slow` and excluded from standard test runs.
|
|
|
|
|
Tests with **low risk** and **execution time >1 second** are now tagged with `@tag :slow` and excluded from standard test runs. These tests are important but not critical for every commit and can be run in nightly CI builds.
|
|
|
|
|
|
|
|
|
|
#### Identified Performance Tests (9 tests)
|
|
|
|
|
#### Tagging Criteria
|
|
|
|
|
|
|
|
|
|
1. **Member LiveView - Boolean Filter Performance** (`test/mv_web/member_live/index_test.exs`)
|
|
|
|
|
- Test: `"boolean filter performance with 150 members"`
|
|
|
|
|
- Duration: ~3.8 seconds
|
|
|
|
|
- Creates 150 members to test filter performance
|
|
|
|
|
**Tagged as `@tag :slow` when:**
|
|
|
|
|
- ✅ Test execution time >1 second
|
|
|
|
|
- ✅ Low risk (not critical for catching regressions in core business logic)
|
|
|
|
|
- ✅ UI/Display tests (formatting, rendering)
|
|
|
|
|
- ✅ Workflow detail tests (not core functionality)
|
|
|
|
|
- ✅ Edge cases with large datasets
|
|
|
|
|
|
|
|
|
|
2. **Group LiveView - Index Performance** (`test/mv_web/live/group_live/index_test.exs`)
|
|
|
|
|
- 2 tests validating efficient page loading and member count calculation
|
|
|
|
|
**NOT tagged when:**
|
|
|
|
|
- ❌ Core CRUD operations (Member/User Create/Update/Destroy)
|
|
|
|
|
- ❌ Basic Authentication/Authorization
|
|
|
|
|
- ❌ Critical Bootstrap (Admin user, system roles)
|
|
|
|
|
- ❌ Email Synchronization
|
|
|
|
|
- ❌ Representative tests per Permission Set + Action
|
|
|
|
|
|
|
|
|
|
3. **Group LiveView - Show Performance** (`test/mv_web/live/group_live/show_test.exs`)
|
|
|
|
|
- 2 tests validating efficient member list loading and slug lookup
|
|
|
|
|
#### Identified Nightly Suite Tests (25 tests)
|
|
|
|
|
|
|
|
|
|
4. **Member LiveView - Membership Fee Status Performance** (`test/mv_web/member_live/index_membership_fee_status_test.exs`)
|
|
|
|
|
- 1 test validating efficient cycle loading without N+1 queries
|
|
|
|
|
**1. Seeds Tests (2 tests) - 18.1s**
|
|
|
|
|
- `"runs successfully and creates basic data"` (9.0s)
|
|
|
|
|
- `"is idempotent when run multiple times"` (9.1s)
|
|
|
|
|
- **Note:** Critical bootstrap tests remain in fast suite
|
|
|
|
|
|
|
|
|
|
5. **Group Integration - Query Performance** (`test/membership/group_integration_test.exs`)
|
|
|
|
|
- 1 test validating N+1 query prevention for group-member relationships
|
|
|
|
|
**2. UserLive.ShowTest (3 tests) - 10.8s**
|
|
|
|
|
- `"mounts successfully with valid user ID"` (4.2s)
|
|
|
|
|
- `"displays linked member when present"` (2.4s)
|
|
|
|
|
- `"redirects to user list when viewing system actor user"` (4.2s)
|
|
|
|
|
|
|
|
|
|
**3. UserLive.IndexTest (5 tests) - 25.0s**
|
|
|
|
|
- `"displays users in a table"` (1.0s)
|
|
|
|
|
- `"initially sorts by email ascending"` (2.2s)
|
|
|
|
|
- `"can sort email descending by clicking sort button"` (3.4s)
|
|
|
|
|
- `"select all automatically checks when all individual users are selected"` (2.0s)
|
|
|
|
|
- `"displays linked member name in user list"` (1.9s)
|
|
|
|
|
|
|
|
|
|
**4. MemberLive.IndexCustomFieldsDisplayTest (3 tests) - 4.9s**
|
|
|
|
|
- `"displays custom field with show_in_overview: true"` (1.6s)
|
|
|
|
|
- `"formats date custom field values correctly"` (1.5s)
|
|
|
|
|
- `"formats email custom field values correctly"` (1.8s)
|
|
|
|
|
|
|
|
|
|
**5. MemberLive.IndexCustomFieldsEdgeCasesTest (3 tests) - 3.6s**
|
|
|
|
|
- `"displays custom field column even when no members have values"` (1.1s)
|
|
|
|
|
- `"displays very long custom field values correctly"` (1.4s)
|
|
|
|
|
- `"handles multiple custom fields with show_in_overview correctly"` (1.2s)
|
|
|
|
|
|
|
|
|
|
**6. RoleLive Tests (7 tests) - 7.7s**
|
|
|
|
|
- `role_live_test.exs`: `"mounts successfully"` (1.5s), `"deletes non-system role"` (2.1s)
|
|
|
|
|
- `role_live/show_test.exs`: 5 tests >1s (mount, display, navigation)
|
|
|
|
|
|
|
|
|
|
**7. MemberAvailableForLinkingTest (1 test) - 1.5s**
|
|
|
|
|
- `"limits results to 10 members even when more exist"` (1.5s)
|
|
|
|
|
|
|
|
|
|
**8. Performance Tests (1 test) - 3.8s**
|
|
|
|
|
- `"boolean filter performance with 150 members"` (3.8s)
|
|
|
|
|
|
|
|
|
|
**Total:** 25 tests, ~77 seconds saved
|
|
|
|
|
|
|
|
|
|
#### Execution Commands
|
|
|
|
|
|
|
|
|
|
@ -116,7 +155,7 @@ just test-fast
|
|
|
|
|
mix test --exclude slow
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
**Performance Tests Only:**
|
|
|
|
|
**Slow/Nightly Tests Only:**
|
|
|
|
|
```bash
|
|
|
|
|
just test-slow
|
|
|
|
|
# or
|
|
|
|
|
@ -132,19 +171,28 @@ mix test
|
|
|
|
|
|
|
|
|
|
#### CI/CD Integration
|
|
|
|
|
|
|
|
|
|
- **Standard CI:** Runs `mix test --exclude slow` for faster feedback loops
|
|
|
|
|
- **Nightly Builds:** Separate pipeline (`nightly-tests`) runs daily at 2 AM and executes all performance tests
|
|
|
|
|
- **Standard CI:** Runs `mix test --exclude slow` for faster feedback loops (~6 minutes)
|
|
|
|
|
- **Nightly Builds:** Separate pipeline runs daily and executes `mix test --only slow` (~1.3 minutes)
|
|
|
|
|
- **Pre-Merge:** Full test suite (`mix test`) runs before merging to main
|
|
|
|
|
- **Manual Execution:** Can be triggered via Drone CLI or Web UI
|
|
|
|
|
|
|
|
|
|
#### Risk Assessment
|
|
|
|
|
|
|
|
|
|
**Risk Level:** ✅ **Very Low**
|
|
|
|
|
|
|
|
|
|
- Performance tests are still executed, just separately
|
|
|
|
|
- Standard test runs are faster, improving developer feedback
|
|
|
|
|
- All tagged tests have **low risk** - they don't catch critical regressions
|
|
|
|
|
- Core functionality remains tested (CRUD, Auth, Bootstrap)
|
|
|
|
|
- Standard test runs are faster (~6 minutes vs ~7.4 minutes)
|
|
|
|
|
- Nightly builds ensure comprehensive coverage
|
|
|
|
|
- No functionality is lost, only execution timing changed
|
|
|
|
|
|
|
|
|
|
**Critical Tests Remain in Fast Suite:**
|
|
|
|
|
- Core CRUD operations (Member/User Create/Update/Destroy)
|
|
|
|
|
- Basic Authentication/Authorization
|
|
|
|
|
- Critical Bootstrap (Admin user, system roles)
|
|
|
|
|
- Email Synchronization
|
|
|
|
|
- Representative Policy tests (one per Permission Set + Action)
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### 3. Critical Test Optimization
|
|
|
|
|
@ -195,19 +243,79 @@ The test loaded **all members** from the database, not just the 2 members from t
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
### 3. Nightly Suite Analysis and Categorization
|
|
|
|
|
|
|
|
|
|
**Date:** 2026-01-28
|
|
|
|
|
**Status:** ✅ Completed
|
|
|
|
|
|
|
|
|
|
#### Analysis Methodology
|
|
|
|
|
|
|
|
|
|
A comprehensive analysis was performed to identify tests suitable for the nightly suite based on:
|
|
|
|
|
- **Execution time:** Tests taking >1 second
|
|
|
|
|
- **Risk assessment:** Tests that don't catch critical regressions
|
|
|
|
|
- **Test category:** UI/Display, workflow details, edge cases
|
|
|
|
|
|
|
|
|
|
#### Test Categorization
|
|
|
|
|
|
|
|
|
|
**🔴 CRITICAL - Must Stay in Fast Suite:**
|
|
|
|
|
- Core Business Logic (Member/User CRUD)
|
|
|
|
|
- Authentication & Authorization Basics
|
|
|
|
|
- Critical Bootstrap (Admin user, system roles)
|
|
|
|
|
- Email Synchronization
|
|
|
|
|
- Representative Policy Tests (one per Permission Set + Action)
|
|
|
|
|
|
|
|
|
|
**🟡 LOW RISK - Moved to Nightly Suite:**
|
|
|
|
|
- Seeds Tests (non-critical: smoke test, idempotency)
|
|
|
|
|
- LiveView Display/Formatting Tests
|
|
|
|
|
- UserLive.ShowTest (core functionality covered by Index/Form)
|
|
|
|
|
- UserLive.IndexTest UI Features (sorting, checkboxes, navigation)
|
|
|
|
|
- RoleLive Tests (role management, not core authorization)
|
|
|
|
|
- MemberLive Custom Fields Display Tests
|
|
|
|
|
- Edge Cases with Large Datasets
|
|
|
|
|
|
|
|
|
|
#### Risk Assessment Summary
|
|
|
|
|
|
|
|
|
|
| Category | Tests | Time Saved | Risk Level | Rationale |
|
|
|
|
|
|----------|-------|------------|------------|-----------|
|
|
|
|
|
| Seeds (non-critical) | 2 | 18.1s | ⚠️ Low | Critical bootstrap tests remain |
|
|
|
|
|
| UserLive.ShowTest | 3 | 10.8s | ⚠️ Low | Core CRUD covered by Index/Form |
|
|
|
|
|
| UserLive.IndexTest (UI) | 5 | 25.0s | ⚠️ Low | UI features, not core functionality |
|
|
|
|
|
| Custom Fields Display | 6 | 8.5s | ⚠️ Low | Formatting tests, visible in code review |
|
|
|
|
|
| RoleLive Tests | 7 | 7.7s | ⚠️ Low | Role management, not authorization |
|
|
|
|
|
| Edge Cases | 1 | 1.5s | ⚠️ Low | Edge case, not critical path |
|
|
|
|
|
| Performance Tests | 1 | 3.8s | ✅ Very Low | Explicit performance validation |
|
|
|
|
|
| **Total** | **25** | **~77s** | **⚠️ Low** | |
|
|
|
|
|
|
|
|
|
|
**Overall Risk:** ⚠️ **Low** - All moved tests have low risk and don't catch critical regressions. Core functionality remains fully tested.
|
|
|
|
|
|
|
|
|
|
#### Tests Excluded from Nightly Suite
|
|
|
|
|
|
|
|
|
|
The following tests were **NOT** moved to nightly suite despite being slow:
|
|
|
|
|
|
|
|
|
|
- **Policy Tests:** Medium risk - kept in fast suite (representative tests remain)
|
|
|
|
|
- **UserLive.FormTest:** Medium risk - core CRUD functionality
|
|
|
|
|
- **Tests <1s:** Don't meet execution time threshold
|
|
|
|
|
- **Critical Bootstrap Tests:** High risk - deployment critical
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
## Current Performance Analysis
|
|
|
|
|
|
|
|
|
|
### Top 20 Slowest Tests (without `:slow`)
|
|
|
|
|
|
|
|
|
|
| Rank | Test | File | Time | % of Top 20 |
|
|
|
|
|
|------|------|------|------|-------------|
|
|
|
|
|
| 1 | `test respects show_in_overview config` | `index_member_fields_display_test.exs` | **~4-6s** (optimized) | ~6-7% |
|
|
|
|
|
| 2 | `test Seeds script is idempotent when run multiple times` | `seeds_test.exs` | 5.0s | 6.2% |
|
|
|
|
|
| 3 | `test sorting functionality initially sorts by email ascending` | `user_live/index_test.exs` | 4.5s | 5.5% |
|
|
|
|
|
| 4 | `test Seeds script runs successfully and creates basic data` | `seeds_test.exs` | 4.3s | 5.3% |
|
|
|
|
|
| 5-20 | Various User LiveView and Policy tests | Multiple files | 2.6-4.2s each | 3-5% each |
|
|
|
|
|
After implementing the nightly suite, the remaining slowest tests are:
|
|
|
|
|
|
|
|
|
|
**Total Top 20:** ~81 seconds (13.2% of total time)
|
|
|
|
|
| Rank | Test | File | Time | Category |
|
|
|
|
|
|------|------|------|------|----------|
|
|
|
|
|
| 1 | `test Critical bootstrap invariants Mitglied system role exists` | `seeds_test.exs` | 6.7s | Critical Bootstrap |
|
|
|
|
|
| 2 | `test Critical bootstrap invariants Admin user has Admin role` | `seeds_test.exs` | 5.0s | Critical Bootstrap |
|
|
|
|
|
| 3 | `test normal_user permission set can read own user record` | `user_policies_test.exs` | 2.6s | Policy Test |
|
|
|
|
|
| 4 | `test normal_user permission set can create member` | `member_policies_test.exs` | 2.5s | Policy Test |
|
|
|
|
|
| 5-20 | Various Policy and LiveView tests | Multiple files | 1.5-2.4s each | Policy/LiveView |
|
|
|
|
|
|
|
|
|
|
**Total Top 20:** ~44 seconds (12% of total time without `:slow`)
|
|
|
|
|
|
|
|
|
|
**Note:** Many previously slow tests (UserLive.IndexTest, UserLive.ShowTest, Display/Formatting tests) are now tagged with `@tag :slow` and excluded from standard runs.
|
|
|
|
|
|
|
|
|
|
### Performance Hotspots Identified
|
|
|
|
|
|
|
|
|
|
@ -329,7 +437,7 @@ The test loaded **all members** from the database, not just the 2 members from t
|
|
|
|
|
| 4 | Additional Isolation | Variable |
|
|
|
|
|
| **Total Potential** | | **23-33 seconds** |
|
|
|
|
|
|
|
|
|
|
**Projected Final Time:** From ~614 seconds to **~580-590 seconds** (~9.5-10 minutes)
|
|
|
|
|
**Projected Final Time:** From ~368 seconds (fast suite) to **~340-350 seconds** (~5.7-5.8 minutes) with remaining optimizations
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
@ -353,8 +461,9 @@ All optimizations maintain test coverage while improving performance:
|
|
|
|
|
- ✅ Seeds tests execute in <20 seconds consistently
|
|
|
|
|
- ✅ No increase in seeds-related deployment failures
|
|
|
|
|
- ✅ No regression in authorization or membership fee bugs
|
|
|
|
|
- ⏳ Top 20 slowest tests: < 60 seconds (currently 81.2s)
|
|
|
|
|
- ⏳ Total execution time (without `:slow`): < 10 minutes (currently 10.2 min)
|
|
|
|
|
- ✅ Top 20 slowest tests: < 60 seconds (currently ~44s)
|
|
|
|
|
- ✅ Total execution time (without `:slow`): < 10 minutes (currently 6.1 min)
|
|
|
|
|
- ⏳ Nightly suite execution time: < 2 minutes (currently ~1.3 min)
|
|
|
|
|
|
|
|
|
|
#### What to Watch For
|
|
|
|
|
|
|
|
|
|
@ -421,18 +530,23 @@ mix test test/mv_web/member_live/index_member_fields_display_test.exs --slowest
|
|
|
|
|
### Test Execution Modes
|
|
|
|
|
|
|
|
|
|
**Fast Tests (Default):**
|
|
|
|
|
- Excludes performance tests (`@tag :slow`)
|
|
|
|
|
- Excludes slow tests (`@tag :slow`)
|
|
|
|
|
- Used for standard development workflow
|
|
|
|
|
- Execution time: ~6 minutes
|
|
|
|
|
- Command: `mix test --exclude slow` or `just test-fast`
|
|
|
|
|
|
|
|
|
|
**Performance Tests:**
|
|
|
|
|
- Explicitly tagged performance tests
|
|
|
|
|
- Run separately or in nightly builds
|
|
|
|
|
**Slow/Nightly Tests:**
|
|
|
|
|
- Tests tagged with `@tag :slow` (25 tests)
|
|
|
|
|
- Low risk, >1 second execution time
|
|
|
|
|
- UI/Display tests, workflow details, edge cases
|
|
|
|
|
- Execution time: ~1.3 minutes
|
|
|
|
|
- Command: `mix test --only slow` or `just test-slow`
|
|
|
|
|
- Run in nightly CI builds
|
|
|
|
|
|
|
|
|
|
**All Tests:**
|
|
|
|
|
- Includes both fast and slow tests
|
|
|
|
|
- Used for comprehensive validation
|
|
|
|
|
- Used for comprehensive validation (pre-merge)
|
|
|
|
|
- Execution time: ~7.4 minutes
|
|
|
|
|
- Command: `mix test` or `just test`
|
|
|
|
|
|
|
|
|
|
### Test Organization
|
|
|
|
|
@ -477,18 +591,33 @@ test/
|
|
|
|
|
4. **Filter queries** - Only load data needed for the test
|
|
|
|
|
5. **Verify coverage** - Ensure optimizations don't reduce test coverage
|
|
|
|
|
|
|
|
|
|
### Performance Test Guidelines
|
|
|
|
|
### Test Tagging Guidelines
|
|
|
|
|
|
|
|
|
|
**Tag as `:slow` when:**
|
|
|
|
|
- Explicitly testing performance characteristics
|
|
|
|
|
- Using large datasets (50+ records)
|
|
|
|
|
- Testing scalability or query optimization
|
|
|
|
|
- Validating N+1 query prevention
|
|
|
|
|
#### Tag as `@tag :slow` when:
|
|
|
|
|
|
|
|
|
|
**Do NOT tag as `:slow` when:**
|
|
|
|
|
- Test is slow due to inefficient setup (fix the setup instead)
|
|
|
|
|
- Test is slow due to bugs (fix the bug instead)
|
|
|
|
|
- It's an integration test (use `@tag :integration` instead)
|
|
|
|
|
1. **Performance Tests:**
|
|
|
|
|
- Explicitly testing performance characteristics
|
|
|
|
|
- Using large datasets (50+ records)
|
|
|
|
|
- Testing scalability or query optimization
|
|
|
|
|
- Validating N+1 query prevention
|
|
|
|
|
|
|
|
|
|
2. **Low-Risk Tests (>1s):**
|
|
|
|
|
- UI/Display/Formatting tests (not critical for every commit)
|
|
|
|
|
- Workflow detail tests (not core functionality)
|
|
|
|
|
- Edge cases with large datasets
|
|
|
|
|
- Show page tests (core functionality covered by Index/Form tests)
|
|
|
|
|
- Non-critical seeds tests (smoke tests, idempotency)
|
|
|
|
|
|
|
|
|
|
#### Do NOT tag as `@tag :slow` when:
|
|
|
|
|
|
|
|
|
|
- ❌ Test is slow due to inefficient setup (fix the setup instead)
|
|
|
|
|
- ❌ Test is slow due to bugs (fix the bug instead)
|
|
|
|
|
- ❌ Core CRUD operations (Member/User Create/Update/Destroy)
|
|
|
|
|
- ❌ Basic Authentication/Authorization
|
|
|
|
|
- ❌ Critical Bootstrap (Admin user, system roles)
|
|
|
|
|
- ❌ Email Synchronization
|
|
|
|
|
- ❌ Representative Policy tests (one per Permission Set + Action)
|
|
|
|
|
- ❌ It's an integration test (use `@tag :integration` instead)
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
@ -506,9 +635,35 @@ test/
|
|
|
|
|
|
|
|
|
|
**Time Saved:** ~21-30 seconds per test run
|
|
|
|
|
|
|
|
|
|
### 2026-01-28: Nightly Suite Implementation
|
|
|
|
|
|
|
|
|
|
**Completed:**
|
|
|
|
|
- ✅ Analyzed all tests for nightly suite candidates
|
|
|
|
|
- ✅ Identified 36 tests with low risk and >1s execution time
|
|
|
|
|
- ✅ Tagged 25 tests with `@tag :slow` for nightly suite
|
|
|
|
|
- ✅ Categorized tests by risk level and execution time
|
|
|
|
|
- ✅ Documented tagging criteria and guidelines
|
|
|
|
|
|
|
|
|
|
**Tests Tagged:**
|
|
|
|
|
- 2 Seeds tests (non-critical) - 18.1s
|
|
|
|
|
- 3 UserLive.ShowTest tests - 10.8s
|
|
|
|
|
- 5 UserLive.IndexTest tests - 25.0s
|
|
|
|
|
- 3 MemberLive.IndexCustomFieldsDisplayTest tests - 4.9s
|
|
|
|
|
- 3 MemberLive.IndexCustomFieldsEdgeCasesTest tests - 3.6s
|
|
|
|
|
- 7 RoleLive tests - 7.7s
|
|
|
|
|
- 1 MemberAvailableForLinkingTest - 1.5s
|
|
|
|
|
- 1 Performance test (already tagged) - 3.8s
|
|
|
|
|
|
|
|
|
|
**Time Saved:** ~77 seconds per test run
|
|
|
|
|
|
|
|
|
|
**Total Optimization Impact:**
|
|
|
|
|
- **Before:** ~445 seconds (7.4 minutes)
|
|
|
|
|
- **After (fast suite):** ~368 seconds (6.1 minutes)
|
|
|
|
|
- **Time saved:** ~77 seconds (17% reduction)
|
|
|
|
|
|
|
|
|
|
**Next Steps:**
|
|
|
|
|
- ⏳ Optimize User LiveView tests (Priority 1)
|
|
|
|
|
- ⏳ Optimize Policy tests (Priority 2)
|
|
|
|
|
- ⏳ Monitor nightly suite execution in CI
|
|
|
|
|
- ⏳ Optimize remaining slow tests (Policy tests, etc.)
|
|
|
|
|
- ⏳ Further optimize Seeds tests (Priority 3)
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
@ -541,3 +696,12 @@ A: Tag it with `@tag :slow` or `@moduletag :slow`. See "Performance Test Guideli
|
|
|
|
|
|
|
|
|
|
**Q: Can I run slow tests locally?**
|
|
|
|
|
A: Yes, use `just test-slow` or `mix test --only slow`. They're excluded from standard runs for faster feedback.
|
|
|
|
|
|
|
|
|
|
**Q: What is the "nightly suite"?**
|
|
|
|
|
A: The "nightly suite" refers to tests tagged with `@tag :slow`. We use a single tag (`:slow`) for both performance tests and tests suitable for nightly execution. All tests tagged with `:slow` are excluded from standard runs and can be executed in nightly CI builds.
|
|
|
|
|
|
|
|
|
|
**Q: Which tests should I tag as `:slow`?**
|
|
|
|
|
A: Tag tests with `@tag :slow` if they: (1) take >1 second, (2) have low risk (not critical for catching regressions), and (3) test UI/Display/Formatting or workflow details. See "Test Tagging Guidelines" section for details.
|
|
|
|
|
|
|
|
|
|
**Q: What if a slow test fails in nightly builds?**
|
|
|
|
|
A: If a test in the nightly suite fails, investigate the failure. If it indicates a critical regression, consider moving it back to the fast suite. If it's a flaky test, fix the test itself.
|
|
|
|
|
|