Minor test refactoring to improve on performance closes #383 #384

Merged

simon merged 23 commits from test-performance-optimization into main 2026-01-29 15:44:01 +01:00

Conversation 0 Commits 23 Files changed 26 +1503 -446

simon commented 2026-01-28 14:51:03 +01:00

Owner

Copy link

Description of the implemented changes

The changes were:

• Bugfixing
• New Feature
• Breaking Change
• Refactoring

Comprehensive test suite optimization: reduce execution time, improve test quality, and establish performance testing infrastructure

What has been changed?

Test Suite Optimization

Seeds Test Suite Reduction

• Reduced from 13 to 4 tests (69% reduction) focusing on critical deployment requirements:
  • Smoke test: seeds run without errors
  • Idempotency test: seeds can be run multiple times
  • Critical bootstrap invariants: Admin user and system roles exist
• Removed redundant tests whose functionality is covered by domain-specific test suites:
  • Member/fee type distribution → covered by membership_fees/*_test.exs
  • Cycle status variation → covered by membership_fees/cycle_generator_test.exs
  • Detailed role configuration → covered by authorization/*_test.exs
• Time saved: ~10-16 seconds per test run (40-50% faster)

Slow Test Suite Implementation

• Tagged 25 tests with @tag :slow or @describetag :slow for nightly execution:
  • 2 Seeds tests (non-critical)
  • 3 UserLive.ShowTest tests
  • 5 UserLive.IndexTest tests
  • 3 MemberLive.IndexCustomFieldsDisplayTest tests
  • 3 MemberLive.IndexCustomFieldsEdgeCasesTest tests
  • 7 RoleLive tests
  • 1 MemberAvailableForLinkingTest
  • 1 Performance test
• Time saved: ~77 seconds per standard CI run
• Total optimization impact: ~98-107 seconds saved per test run (17% reduction)

CI/CD Pipeline Updates

Standard CI Pipeline

• Changed to run fast tests only: mix test --exclude slow instead of mix test
• Execution time: Reduced from ~7.4 minutes to ~6.1 minutes
• Updated ci-dev command: Now uses test-fast instead of test

New Nightly CI Pipeline

• Added nightly-tests pipeline that runs daily at 2 AM (cron) and can be manually triggered
• Runs all tests (mix test) for comprehensive coverage including slow/performance tests
• Step renamed from test-slow to test-all to accurately reflect functionality
• Execution time: ~7.4 minutes (full test suite)

Developer Workflow Improvements

Justfile Commands

• Added test-fast: Runs only fast tests (mix test --exclude slow)
• Added test-slow: Runs only slow/performance tests (mix test --only slow)
• Added test-all: Runs all tests (mix test)
• Updated ci-dev: Now uses test-fast for faster feedback

Test Quality Improvements

UserLive Test Refactoring

• Removed framework functionality tests (~15-20 tests):
  • Navigation tests (Phoenix LiveView routing)
  • Translation tests (Gettext functionality)
  • Basic HTML rendering tests
  • Sort direction icon tests
• Focused on business logic:
  • System Actor handling
  • Member linking functionality
  • Password authentication status
  • User deletion workflow
• Fixed test quality issues:
  • Checkbox selection test now properly asserts select_all checked state
  • Delete test now verifies user is actually deleted from database
  • Password assertion removed bcrypt prefix check (implementation-independent)

Performance Test Enhancements

• Added query count assertions to GroupLive.ShowTest and GroupLive.IndexTest:
  • Uses Telemetry to count database queries
  • Verifies N+1 query prevention with measurable assertions
  • Includes proper handler cleanup with unique IDs
• Fixed test tagging: Changed @moduletag :slow to @describetag :slow in GroupLive.ShowTest to prevent navigation tests from being incorrectly tagged

Documentation

New Analysis Documents

• docs/test-performance-optimization.md (712 lines):

  • Comprehensive overview of test performance optimizations
  • Test execution modes and organization
  • Tagging guidelines for slow tests
  • Benchmarking best practices
  • Risk assessment and optimization impact

• docs/test-optimization-user-liveview-analysis.md (524 lines):

  • Detailed analysis of User LiveView test performance
  • Framework functionality over-testing identification
  • Test categorization and optimization recommendations
  • Performance metrics and time savings estimates

• docs/test-optimization-policy-tests-analysis.md (868 lines):

  • Deep analysis of policy test suite (74 tests, ~152s total)
  • Identified optimization opportunities:
    • Redundant fixture creation
    • Framework functionality over-testing
    • Test duplication across permission sets
  • Estimated time savings: 5-8 seconds
  • Ready for future implementation

Updated Documentation

• CODE_GUIDELINES.md:

  • Added performance test guidelines
  • Clarified @describetag vs @moduletag usage
  • Added guidance for measurable performance assertions
  • Updated test execution commands

• test/seeds_test.exs:

  • Added comprehensive @moduledoc explaining test reduction rationale
  • Documented removed tests and their coverage mapping

Test Infrastructure

Test Helper Updates

• Updated ExUnit configuration to show slowest tests automatically
• Improved test organization with better tagging structure

Definition of Done

Code Quality

• No new technical depths
• Linting passed
• Documentation is added were needed

Accessibility

• New elements are properly defined with html-tags
• Colour contrast follows WCAG criteria
• Aria labels are added when needed
• Everything is accessible by keyboard
• Tab-Order is comprehensible
• All interactive elements have a visible focus

Testing

• Tests for new code are written
• All tests pass
• axe-core dev tools show no critical or major issues

Additional Notes

Performance Impact Summary

Metric	Before	After	Improvement
Standard CI Time	~7.4 min	~6.1 min	17% faster
Seeds Tests	13 tests (~24-30s)	4 tests (~13-17s)	40-50% faster
Slow Tests Tagged	0	25 tests	~77s excluded from standard CI
Total Time Saved	-	-	~98-107s per run

Future Optimization Opportunities

The following analysis documents identify additional optimization opportunities for future work:

1. Policy Tests Suite (docs/test-optimization-policy-tests-analysis.md):

   • 74 tests taking ~152 seconds total
   • Potential savings: 5-8 seconds through fixture sharing and test deduplication
   • Framework functionality over-testing identified
   • Ready for implementation decision

2. User LiveView Tests (docs/test-optimization-user-liveview-analysis.md):

   • Additional framework tests could be removed
   • Shared fixture opportunities identified
   • Estimated additional savings: 15-20 seconds

Code Review Fixes

This PR addresses feedback from multiple code reviews:

1. Commit 67e06e12ce fixes:

   • Fixed @moduletag :slow scope issue in GroupLive.ShowTest
   • Clarified nightly CI runs all tests, not just slow tests
   • Added performance assertions to performance tests

2. Commit 91f8bb03bc fixes:

   • Fixed checkbox selection test assertion
   • Fixed delete functionality test to verify actual deletion
   • Removed implementation-dependent password assertion

Breaking Changes

None - All changes are backward compatible:

• Test coverage remains the same (all tests still run in nightly CI)
• No production code changes
• Only test organization and execution timing changed

Migration Notes

Developers should:

• Use just test-fast or mix test --exclude slow for faster local feedback
• Use just test-all or mix test before committing to ensure full coverage
• Tag new performance tests with @tag :slow or @describetag :slow (not @moduletag)
• Include measurable assertions (query counts, timing) in performance tests

## Description of the implemented changes The changes were: - [x] Bugfixing - [ ] New Feature - [ ] Breaking Change - [x] Refactoring **Comprehensive test suite optimization: reduce execution time, improve test quality, and establish performance testing infrastructure** ## What has been changed? ### Test Suite Optimization #### Seeds Test Suite Reduction - **Reduced from 13 to 4 tests** (69% reduction) focusing on critical deployment requirements: - Smoke test: seeds run without errors - Idempotency test: seeds can be run multiple times - Critical bootstrap invariants: Admin user and system roles exist - **Removed redundant tests** whose functionality is covered by domain-specific test suites: - Member/fee type distribution → covered by `membership_fees/*_test.exs` - Cycle status variation → covered by `membership_fees/cycle_generator_test.exs` - Detailed role configuration → covered by `authorization/*_test.exs` - **Time saved:** ~10-16 seconds per test run (40-50% faster) #### Slow Test Suite Implementation - **Tagged 25 tests with `@tag :slow` or `@describetag :slow`** for nightly execution: - 2 Seeds tests (non-critical) - 3 UserLive.ShowTest tests - 5 UserLive.IndexTest tests - 3 MemberLive.IndexCustomFieldsDisplayTest tests - 3 MemberLive.IndexCustomFieldsEdgeCasesTest tests - 7 RoleLive tests - 1 MemberAvailableForLinkingTest - 1 Performance test - **Time saved:** ~77 seconds per standard CI run - **Total optimization impact:** ~98-107 seconds saved per test run (17% reduction) ### CI/CD Pipeline Updates #### Standard CI Pipeline - **Changed to run fast tests only:** `mix test --exclude slow` instead of `mix test` - **Execution time:** Reduced from ~7.4 minutes to ~6.1 minutes - **Updated `ci-dev` command:** Now uses `test-fast` instead of `test` #### New Nightly CI Pipeline - **Added `nightly-tests` pipeline** that runs daily at 2 AM (cron) and can be manually triggered - **Runs all tests** (`mix test`) for comprehensive coverage including slow/performance tests - **Step renamed** from `test-slow` to `test-all` to accurately reflect functionality - **Execution time:** ~7.4 minutes (full test suite) ### Developer Workflow Improvements #### Justfile Commands - **Added `test-fast`:** Runs only fast tests (`mix test --exclude slow`) - **Added `test-slow`:** Runs only slow/performance tests (`mix test --only slow`) - **Added `test-all`:** Runs all tests (`mix test`) - **Updated `ci-dev`:** Now uses `test-fast` for faster feedback ### Test Quality Improvements #### UserLive Test Refactoring - **Removed framework functionality tests** (~15-20 tests): - Navigation tests (Phoenix LiveView routing) - Translation tests (Gettext functionality) - Basic HTML rendering tests - Sort direction icon tests - **Focused on business logic:** - System Actor handling - Member linking functionality - Password authentication status - User deletion workflow - **Fixed test quality issues:** - Checkbox selection test now properly asserts `select_all` checked state - Delete test now verifies user is actually deleted from database - Password assertion removed bcrypt prefix check (implementation-independent) #### Performance Test Enhancements - **Added query count assertions** to `GroupLive.ShowTest` and `GroupLive.IndexTest`: - Uses Telemetry to count database queries - Verifies N+1 query prevention with measurable assertions - Includes proper handler cleanup with unique IDs - **Fixed test tagging:** Changed `@moduletag :slow` to `@describetag :slow` in `GroupLive.ShowTest` to prevent navigation tests from being incorrectly tagged ### Documentation #### New Analysis Documents - **`docs/test-performance-optimization.md`** (712 lines): - Comprehensive overview of test performance optimizations - Test execution modes and organization - Tagging guidelines for slow tests - Benchmarking best practices - Risk assessment and optimization impact - **`docs/test-optimization-user-liveview-analysis.md`** (524 lines): - Detailed analysis of User LiveView test performance - Framework functionality over-testing identification - Test categorization and optimization recommendations - Performance metrics and time savings estimates - **`docs/test-optimization-policy-tests-analysis.md`** (868 lines): - Deep analysis of policy test suite (74 tests, ~152s total) - Identified optimization opportunities: - Redundant fixture creation - Framework functionality over-testing - Test duplication across permission sets - Estimated time savings: 5-8 seconds - Ready for future implementation #### Updated Documentation - **`CODE_GUIDELINES.md`:** - Added performance test guidelines - Clarified `@describetag` vs `@moduletag` usage - Added guidance for measurable performance assertions - Updated test execution commands - **`test/seeds_test.exs`:** - Added comprehensive `@moduledoc` explaining test reduction rationale - Documented removed tests and their coverage mapping ### Test Infrastructure #### Test Helper Updates - **Updated ExUnit configuration** to show slowest tests automatically - **Improved test organization** with better tagging structure ## Definition of Done ### Code Quality - [x] No new technical depths - [x] Linting passed - [x] Documentation is added were needed ### Accessibility - [x] New elements are properly defined with html-tags - [x] Colour contrast follows WCAG criteria - [x] Aria labels are added when needed - [x] Everything is accessible by keyboard - [x] Tab-Order is comprehensible - [x] All interactive elements have a visible focus ### Testing - [x] Tests for new code are written - [x] All tests pass - [x] axe-core dev tools show no critical or major issues ## Additional Notes ### Performance Impact Summary | Metric | Before | After | Improvement | |--------|--------|-------|-------------| | **Standard CI Time** | ~7.4 min | ~6.1 min | **17% faster** | | **Seeds Tests** | 13 tests (~24-30s) | 4 tests (~13-17s) | **40-50% faster** | | **Slow Tests Tagged** | 0 | 25 tests | **~77s excluded from standard CI** | | **Total Time Saved** | - | - | **~98-107s per run** | ### Future Optimization Opportunities The following analysis documents identify additional optimization opportunities for future work: 1. **Policy Tests Suite** (`docs/test-optimization-policy-tests-analysis.md`): - 74 tests taking ~152 seconds total - Potential savings: 5-8 seconds through fixture sharing and test deduplication - Framework functionality over-testing identified - Ready for implementation decision 2. **User LiveView Tests** (`docs/test-optimization-user-liveview-analysis.md`): - Additional framework tests could be removed - Shared fixture opportunities identified - Estimated additional savings: 15-20 seconds ### Code Review Fixes This PR addresses feedback from multiple code reviews: 1. **Commit `67e06e12ce` fixes:** - Fixed `@moduletag :slow` scope issue in `GroupLive.ShowTest` - Clarified nightly CI runs all tests, not just slow tests - Added performance assertions to performance tests 2. **Commit `91f8bb03bc` fixes:** - Fixed checkbox selection test assertion - Fixed delete functionality test to verify actual deletion - Removed implementation-dependent password assertion ### Breaking Changes **None** - All changes are backward compatible: - Test coverage remains the same (all tests still run in nightly CI) - No production code changes - Only test organization and execution timing changed ### Migration Notes Developers should: - Use `just test-fast` or `mix test --exclude slow` for faster local feedback - Use `just test-all` or `mix test` before committing to ensure full coverage - Tag new performance tests with `@tag :slow` or `@describetag :slow` (not `@moduletag`) - Include measurable assertions (query counts, timing) in performance tests

simon added 11 commits 2026-01-28 14:51:04 +01:00

refactor: improve seeds tests performance by reducing complexity f9403c1da9

style: fix formatting fce01ddf83

refactor: move slow performance tests to extra test suite 67e06e12ce

chore: allow manual nightly-tests pipeline run 858a0fc0d0

refactor: apply review comments 6efad280bd

test: optimize single test and update docs 15d328afbf

refactor: remove tests against basic framework functionalities 91f8bb03bc

docs: add performance analysis on policy tests eb2b2436be

test: move slow and less critical tests to nightly suite 050ca4a13c

refactor: apply review comments ea3bdcaa65

refactor: implement more review comments c3ad8894b0

simon changed title from Minor test refactoring to improve on performance to Minor test refactoring to improve on performance closes #383 2026-01-28 14:51:23 +01:00

simon added 1 commit 2026-01-28 14:55:09 +01:00

chore: disable test performance output again 3f0dc868c9

simon added 1 commit 2026-01-28 15:04:32 +01:00

chore: update drone nightly pipeline 25da6a6820

simon added 2 commits 2026-01-29 13:02:41 +01:00

test: restore removed tests including optimizations 0b29fbbd21

docs: update coding guidelines 1019914d50

simon added 1 commit 2026-01-29 14:30:21 +01:00

chore: change pr merge workflow 17974d7a12

simon added 1 commit 2026-01-29 14:39:40 +01:00

test: fix tests 0a1b52d978

simon added 1 commit 2026-01-29 14:42:53 +01:00

chore: remove pr trigger again dddad69e88

simon added 1 commit 2026-01-29 14:49:47 +01:00

fix: make sure all tests run bb7e3cbe77

simon added 2 commits 2026-01-29 15:22:28 +01:00

fix: select all checkbox handling 124ab295a6

feix: optimize queries for groups 9feb6a47aa

simon force-pushed test-performance-optimization from 9feb6a47aa to b4adf63e83 2026-01-29 15:23:22 +01:00 Compare

simon added 1 commit 2026-01-29 15:27:01 +01:00

fix: credo error 9b314a9806

simon added 1 commit 2026-01-29 15:34:22 +01:00

docs: consolidate test performance docs 709cf010c6

simon merged commit ca88a230b9 into main 2026-01-29 15:44:01 +01:00

simon referenced this pull request from a commit 2026-01-29 15:44:04 +01:00

Merge pull request 'Minor test refactoring to improve on performance closes #383' (#384) from test-performance-optimization into main

moritz deleted branch test-performance-optimization 2026-01-29 16:00:04 +01:00

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

enhancement

New feature

  

help wanted

Need some help

  

high priority

  

invalid

Something is wrong

  

L

5 h

  

low priority

  

M

3 h

  

medium priority

  

needs refinement

  

optional

  

question

More information is needed

  

S

1 h

  

UX research

  

wontfix

This won't be fixed

No labels

bug

duplicate

enhancement

help wanted

high priority

invalid

L

low priority

M

medium priority

needs refinement

optional

question

S

UX research

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: local-it/mitgliederverwaltung#384

No description provided.