chore: change pr merge workflow
Some checks reported errors
continuous-integration/drone/push Build is passing
continuous-integration/drone/pr Build was killed

This commit is contained in:
Simon 2026-01-29 14:30:09 +01:00
parent 1019914d50
commit 17974d7a12
Signed by: simon
GPG key ID: 40E7A58C4AA1EDB2
4 changed files with 193 additions and 88 deletions

View file

@ -27,7 +27,7 @@ This document provides a comprehensive overview of test performance optimization
| Seeds tests reduction | 13 → 4 tests | ~10-16s | ✅ Completed |
| Performance tests tagging | 9 tests | ~3-4s per run | ✅ Completed |
| Critical test query filtering | 1 test | ~8-10s | ✅ Completed |
| Nightly suite tagging | 25 tests | ~77s per run | ✅ Completed |
| Full test suite via promotion | 25 tests | ~77s per run | ✅ Completed |
| **Total Saved** | | **~98-107s** | |
---
@ -80,14 +80,14 @@ Critical deployment requirements are still covered:
---
### 2. Nightly Suite Implementation (`@tag :slow`)
### 2. Full Test Suite via Promotion (`@tag :slow`)
**Date:** 2026-01-28
**Status:** ✅ Completed
#### What Changed
Tests with **low risk** and **execution time >1 second** are now tagged with `@tag :slow` and excluded from standard test runs. These tests are important but not critical for every commit and can be run in nightly CI builds.
Tests with **low risk** and **execution time >1 second** are now tagged with `@tag :slow` and excluded from standard test runs. These tests are important but not critical for every commit and are run via promotion before merging to `main`.
#### Tagging Criteria
@ -105,7 +105,7 @@ Tests with **low risk** and **execution time >1 second** are now tagged with `@t
- ❌ Email Synchronization
- ❌ Representative tests per Permission Set + Action
#### Identified Nightly Suite Tests (25 tests)
#### Identified Tests for Full Test Suite (25 tests)
**1. Seeds Tests (2 tests) - 18.1s**
- `"runs successfully and creates basic data"` (9.0s)
@ -155,7 +155,7 @@ just test-fast
mix test --exclude slow
```
**Slow/Nightly Tests Only:**
**Slow Tests Only:**
```bash
just test-slow
# or
@ -171,10 +171,10 @@ mix test
#### CI/CD Integration
- **Standard CI:** Runs `mix test --exclude slow` for faster feedback loops (~6 minutes)
- **Nightly Builds:** Separate pipeline runs daily and executes `mix test` (all tests, including slow) for comprehensive coverage (~7.4 minutes)
- **Pre-Merge:** Full test suite (`mix test`) runs before merging to main
- **Manual Execution:** Can be triggered via Drone CLI or Web UI
- **Standard CI (`check-fast`):** Runs `mix test --exclude slow --exclude ui` for faster feedback loops (~6 minutes)
- **Full Test Suite (`check-full`):** Triggered via promotion before merge, executes `mix test` (all tests, including slow and UI) for comprehensive coverage (~7.4 minutes)
- **Pre-Merge:** Full test suite (`mix test`) runs via promotion before merging to main
- **Manual Execution:** Promote build to `production` in Drone CI to trigger full test suite
#### Risk Assessment
@ -183,7 +183,7 @@ mix test
- All tagged tests have **low risk** - they don't catch critical regressions
- Core functionality remains tested (CRUD, Auth, Bootstrap)
- Standard test runs are faster (~6 minutes vs ~7.4 minutes)
- Nightly builds ensure comprehensive coverage
- Full test suite runs via promotion before merge ensures comprehensive coverage
- No functionality is lost, only execution timing changed
**Critical Tests Remain in Fast Suite:**
@ -243,14 +243,14 @@ The test loaded **all members** from the database, not just the 2 members from t
---
### 3. Nightly Suite Analysis and Categorization
### 3. Full Test Suite Analysis and Categorization
**Date:** 2026-01-28
**Status:** ✅ Completed
#### Analysis Methodology
A comprehensive analysis was performed to identify tests suitable for the nightly suite based on:
A comprehensive analysis was performed to identify tests suitable for the full test suite (via promotion) based on:
- **Execution time:** Tests taking >1 second
- **Risk assessment:** Tests that don't catch critical regressions
- **Test category:** UI/Display, workflow details, edge cases
@ -264,7 +264,7 @@ A comprehensive analysis was performed to identify tests suitable for the nightl
- Email Synchronization
- Representative Policy Tests (one per Permission Set + Action)
**🟡 LOW RISK - Moved to Nightly Suite:**
**🟡 LOW RISK - Moved to Full Test Suite (via Promotion):**
- Seeds Tests (non-critical: smoke test, idempotency)
- LiveView Display/Formatting Tests
- UserLive.ShowTest (core functionality covered by Index/Form)
@ -288,9 +288,9 @@ A comprehensive analysis was performed to identify tests suitable for the nightl
**Overall Risk:** ⚠️ **Low** - All moved tests have low risk and don't catch critical regressions. Core functionality remains fully tested.
#### Tests Excluded from Nightly Suite
#### Tests Excluded from Full Test Suite
The following tests were **NOT** moved to nightly suite despite being slow:
The following tests were **NOT** moved to full test suite (via promotion) despite being slow:
- **Policy Tests:** Medium risk - kept in fast suite (representative tests remain)
- **UserLive.FormTest:** Medium risk - core CRUD functionality
@ -303,7 +303,7 @@ The following tests were **NOT** moved to nightly suite despite being slow:
### Top 20 Slowest Tests (without `:slow`)
After implementing the nightly suite, the remaining slowest tests are:
After implementing the full test suite via promotion, the remaining slowest tests are:
| Rank | Test | File | Time | Category |
|------|------|------|------|----------|
@ -463,7 +463,7 @@ All optimizations maintain test coverage while improving performance:
- ✅ No regression in authorization or membership fee bugs
- ✅ Top 20 slowest tests: < 60 seconds (currently ~44s)
- ✅ Total execution time (without `:slow`): < 10 minutes (currently 6.1 min)
- ⏳ Nightly suite execution time: < 2 minutes (currently ~1.3 min)
- ⏳ Slow tests execution time: < 2 minutes (currently ~1.3 min)
#### What to Watch For
@ -543,10 +543,11 @@ mix test test/mv_web/member_live/index_member_fields_display_test.exs --slowest
- Command: `mix test --only slow` or `just test-slow`
- Excluded from standard CI runs
**Nightly CI Builds:**
**Full Test Suite (via Promotion):**
- Triggered by promoting a build to `production` in Drone CI
- Runs all tests (`mix test`) for comprehensive coverage
- Execution time: ~7.4 minutes
- Ensures full test coverage including slow/performance tests
- Required before merging to `main` (enforced via branch protection)
**All Tests:**
- Includes both fast and slow tests
@ -636,16 +637,16 @@ test/
- ✅ Optimized critical test with query filtering
- ✅ Created slow test suite infrastructure
- ✅ Updated CI/CD to exclude slow tests from standard runs
- ✅ Added nightly CI pipeline for slow tests
- ✅ Added promotion-based full test suite pipeline (`check-full`)
**Time Saved:** ~21-30 seconds per test run
### 2026-01-28: Nightly Suite Implementation
### 2026-01-28: Full Test Suite via Promotion Implementation
**Completed:**
- ✅ Analyzed all tests for nightly suite candidates
- ✅ Analyzed all tests for full test suite candidates
- ✅ Identified 36 tests with low risk and >1s execution time
- ✅ Tagged 25 tests with `@tag :slow` for nightly suite
- ✅ Tagged 25 tests with `@tag :slow` for full test suite (via promotion)
- ✅ Categorized tests by risk level and execution time
- ✅ Documented tagging criteria and guidelines
@ -667,7 +668,7 @@ test/
- **Time saved:** ~77 seconds (17% reduction)
**Next Steps:**
- ⏳ Monitor nightly suite execution in CI
- ⏳ Monitor full test suite execution via promotion in CI
- ⏳ Optimize remaining slow tests (Policy tests, etc.)
- ⏳ Further optimize Seeds tests (Priority 3)
@ -702,11 +703,11 @@ A: Tag it with `@tag :slow` for individual tests or `@describetag :slow` for des
**Q: Can I run slow tests locally?**
A: Yes, use `just test-slow` or `mix test --only slow`. They're excluded from standard runs for faster feedback.
**Q: What is the "nightly suite"?**
A: The "nightly suite" refers to the nightly CI pipeline that runs **all tests** (`mix test`), including slow tests. Tests tagged with `@tag :slow` or `@describetag :slow` are excluded from standard CI runs for faster feedback, but are included in the nightly full test suite for comprehensive coverage.
**Q: What is the "full test suite"?**
A: The full test suite runs **all tests** (`mix test`), including slow and UI tests. Tests tagged with `@tag :slow` or `@describetag :slow` are excluded from standard CI runs (`check-fast`) for faster feedback, but are included when promoting a build to `production` (`check-full`) before merging to `main`.
**Q: Which tests should I tag as `:slow`?**
A: Tag tests with `@tag :slow` if they: (1) take >1 second, (2) have low risk (not critical for catching regressions), and (3) test UI/Display/Formatting or workflow details. See "Test Tagging Guidelines" section for details.
**Q: What if a slow test fails in nightly builds?**
A: If a test in the nightly suite fails, investigate the failure. If it indicates a critical regression, consider moving it back to the fast suite. If it's a flaky test, fix the test itself.
**Q: What if a slow test fails in the full test suite?**
A: If a test in the full test suite fails, investigate the failure. If it indicates a critical regression, consider moving it back to the fast suite. If it's a flaky test, fix the test itself. The merge will be blocked until all tests pass.