# Development Progress Log **Project:** Mila - Membership Management System **Repository:** https://git.local-it.org/local-it/mitgliederverwaltung **License:** AGPLv3 **Status:** Early Development (⚠️ Not Production Ready) --- ## Table of Contents 1. [Project Overview](#project-overview) 2. [Setup and Foundation](#setup-and-foundation) 3. [Major Features Implementation](#major-features-implementation) 4. [Implementation Decisions](#implementation-decisions) 5. [Build and Deployment](#build-and-deployment) 6. [Testing Strategy](#testing-strategy) 7. [Common Issues and Solutions](#common-issues-and-solutions) 8. [Future Improvements](#future-improvements) 9. [Team Knowledge Base](#team-knowledge-base) --- ## Project Overview ### Vision Simple, usable, self-hostable membership management for small to mid-sized clubs. ### Philosophy *"Software should help people spend less time on administration and more time on their community."* ### Core Principles - ✅ **Simple:** Focused on essential club needs - ✅ **Usable:** Clean, accessible UI for everyday volunteers - ✅ **Flexible:** Customizable data fields, role-based permissions - ✅ **Open:** 100% free and open source, no vendor lock-in - ✅ **Self-hostable:** Full control over data and deployment ### Target Users - Small to mid-sized clubs - Volunteer administrators (non-technical) - Club members (self-service access) --- ## Setup and Foundation ### Initial Project Setup For **current setup instructions**, see [`README.md`](../README.md#-quick-start-development). **Historical context:** #### 1. Phoenix Project Initialization (Sprint 0) ```bash mix phx.new mv --no-ecto --no-mailer ``` **Reasoning:** - `--no-ecto`: Using Ash Framework with AshPostgres instead - `--no-mailer`: Added Swoosh later for better control #### 2. Technology Choices **For complete tech stack details, see [`CODE_GUIDELINES.md`](../CODE_GUIDELINES.md#project-context).** **Key decisions:** - **Elixir 1.18.3 + OTP 27**: Latest stable versions for performance - **Ash Framework 3.0**: Declarative resource layer, reduces boilerplate - **Phoenix LiveView 1.1**: Real-time UI without JavaScript complexity - **Tailwind CSS 4.0**: Utility-first styling with custom build - **PostgreSQL 17**: Advanced features (full-text search, JSONB, citext) - **Bandit**: Modern HTTP server, better than Cowboy for LiveView #### 3. Version Management (asdf) **Tool:** asdf 0.16.5 for consistent environments across team **Versions pinned in `.tool-versions`:** - Elixir 1.18.3-otp-27 - Erlang 27.3.4 - Just 1.43.0 #### 4. Database Setup **PostgreSQL Extensions:** ```sql CREATE EXTENSION IF NOT EXISTS "uuid-ossp"; -- UUID generation CREATE EXTENSION IF NOT EXISTS "citext"; -- Case-insensitive text ``` **Migration Strategy:** ```bash mix ash.codegen --name # Generate from Ash resources mix ash.migrate # Apply migrations ``` **Reasoning:** Ash generates migrations from resource definitions, ensuring schema matches code. #### 5. Development Workflow (Just) Chose **Just** over Makefile for: - Better error messages - Cleaner syntax - Cross-platform compatibility **Core commands:** See [README.md](../README.md#-development) --- ## Major Features Implementation ### Sprint History & Key Pull Requests Based on closed PRs from https://git.local-it.org/local-it/mitgliederverwaltung/pulls?state=closed: #### Phase 1: Foundation (Sprint 0-2) **Sprint 0 - Vorarbeit** - Initial project setup - Technology stack decisions - Repository structure **Sprint 2 - 07.05 - 28.05** - Basic Phoenix setup - Initial database schema - Development environment configuration #### Phase 2: Core Features (Sprint 3-5) **Sprint 3 - 28.05 - 09.07** - Member CRUD operations - Basic property system - Initial UI with Tailwind CSS **Sprint 4 - 09.07 - 30.07** - Property types implementation - Data validation - Error handling improvements **Sprint 5 - 31.07 - 11.09** **PR #138:** *Customize login screen and members as landing page* (closes #68, #137) - Custom login UI with DaisyUI - Members page as default landing - Improved navigation flow **PR #139:** *Added PR and issue templates* (closes #129) - GitHub/GitLab issue templates - PR template for consistent reviews - Contribution guidelines **PR #147:** *Add seed data for members* - Comprehensive seed data - Test users and members - Property type examples #### Phase 3: Search & Navigation (Sprint 6) **Sprint 6 - 11.09 - 02.10** **PR #163:** *Implement full-text search for members* (closes #11) 🔍 - PostgreSQL full-text search with tsvector - Weighted search fields (names: A, email/notes: B, contact: C) - GIN index for performance - Auto-updating trigger - Migration: `20250912085235_AddSearchVectorToMembers.exs` ```elixir # Search implementation highlights attribute :search_vector, AshPostgres.Tsvector, writable?: false, public?: false, select_by_default?: false ``` **Key learnings:** - Simple lexer used (no German stemming initially) - Weighted fields improve relevance - GIN index essential for performance #### Phase 4: Sorting & User Management (Sprint 7) **Sprint 7 - 02.10 - 23.10** **PR #166:** *Sorting header for members list* (closes #152, #175) - Sortable table headers component - Multi-column sorting support - Visual indicators for sort direction - Accessibility improvements (ARIA labels) **PR #172:** *Create logical link between users and members* (closes #164) - Optional 1:1 relationship (0..1 ↔ 0..1) - User `belongs_to` Member - Member `has_one` User - Foundation for email sync feature - Migration: `20250926164519_member_relation.exs` **PR #148:** *Fix error when deleting members* - Cascade delete handling - Proper foreign key constraints - Error message improvements **PR #173:** *Link to user data from profile button* (closes #170) - Profile navigation improvements - User-member relationship display - Better UX for linked accounts **PR #178:** *Polish README* (closes #158) - Updated documentation - Better onboarding instructions - Screenshots and examples #### Phase 5: Email Synchronization (Sprint 8) **Sprint 8 - 23.10 - 13.11** **PR #181:** *Sync email between user and member* (closes #167) ✉️ - Bidirectional email synchronization between User and Member - User.email as source of truth on linking - Custom Ash changes with conditional execution - Complex validation logic to prevent conflicts - Migration: `20251016130855_add_constraints_for_user_member_and_property.exs` **See:** [`docs/email-sync.md`](email-sync.md) for complete sync rules and decision tree. --- ## Implementation Decisions ### Architecture Patterns #### 1. Ash Framework Over Traditional Phoenix **Decision:** Use Ash Framework as the primary data layer instead of traditional Ecto contexts. **Reasoning:** - **Declarative resource definitions** reduce boilerplate - **Built-in authorization** with policies - **Type safety** with calculations and aggregates - **Code generation** for migrations **Trade-offs:** - Steeper learning curve - Less common in Phoenix community - Newer ecosystem (fewer resources) - More opinionated structure **Outcome:** - ✅ Faster feature development - ✅ Consistent API across resources - ⚠️ Requires team training #### 2. Domain-Driven Design **Decision:** Organize by business domains (Accounts, Membership) rather than technical layers. **Reasoning:** - Clear separation of concerns - Business logic separate from web layer - Scalable for future domains (payments, communications) **For detailed project structure, see [`CODE_GUIDELINES.md`](../CODE_GUIDELINES.md#11-project-structure).** #### 3. Bidirectional Email Sync **Problem:** Users and Members can exist independently, but when linked, emails must stay synchronized. **Solution:** Custom Ash changes with conditional execution **Why not simpler approaches?** - ❌ Single email table: Too restrictive (members without users need emails) - ❌ Always sync: Performance concerns, unnecessary for unlinked entities - ❌ Manual sync: Error-prone, inconsistent - ✅ Conditional sync with validations: Flexible, safe, performant **Complete documentation:** See [`docs/email-sync.md`](email-sync.md) for decision tree and sync rules. #### 4. Property System (EAV Pattern) **Implementation:** Entity-Attribute-Value pattern with union types ```elixir # Property Type defines schema defmodule Mv.Membership.PropertyType do attribute :name, :string # "Membership Number" attribute :value_type, :atom # :string, :integer, :boolean, :date, :email attribute :immutable, :boolean # Can't change after creation attribute :required, :boolean # All members must have this end # Property stores values defmodule Mv.Membership.Property do attribute :value, :union, # Polymorphic value storage constraints: [ types: [ string: [type: :string], integer: [type: :integer], boolean: [type: :boolean], date: [type: :date], email: [type: Mv.Membership.Email] ] ] belongs_to :member belongs_to :property_type end ``` **Reasoning:** - Clubs need different custom fields - No schema migrations for new fields - Type safety with union types - Centralized property management **Constraints:** - One property per type per member (composite unique index) - Properties deleted with member (CASCADE) - Property types protected if in use (RESTRICT) #### 5. Authentication Strategy **Multi-Strategy Authentication:** ```elixir authentication do strategies do # Password-based password :password do identity_field :email hash_provider AshAuthentication.BcryptProvider end # OIDC (Rauthy) oidc :rauthy do client_id Mv.Secrets base_url Mv.Secrets client_secret Mv.Secrets end end end ``` **Reasoning:** - Flexibility: Clubs choose authentication method - Self-hosting: OIDC with Rauthy (open source) - Fallback: Password auth for non-SSO users - Security: bcrypt for password hashing **Token Management:** - Store all tokens (store_all_tokens? true) - JWT-based sessions - Token revocation support #### 6. UI Framework Choice **Tailwind CSS + DaisyUI** **Reasoning:** - **Tailwind:** Utility-first, no custom CSS - **DaisyUI:** Pre-built components, consistent design - **Heroicons:** Icon library, inline SVG - **Phoenix LiveView:** Server-rendered, minimal JavaScript **Trade-offs:** - Larger HTML (utility classes) - Learning curve for utility-first CSS - ✅ Faster development - ✅ Consistent styling - ✅ Mobile-responsive out of the box #### 7. Full-Text Search Implementation **PostgreSQL tsvector + GIN Index** ```sql -- Auto-updating trigger CREATE FUNCTION members_search_vector_trigger() RETURNS trigger AS $$ BEGIN NEW.search_vector := setweight(to_tsvector('simple', coalesce(NEW.first_name, '')), 'A') || setweight(to_tsvector('simple', coalesce(NEW.last_name, '')), 'A') || setweight(to_tsvector('simple', coalesce(NEW.email, '')), 'B') || setweight(to_tsvector('simple', coalesce(NEW.notes, '')), 'B') || setweight(to_tsvector('simple', coalesce(NEW.city, '')), 'C') || -- ... more fields RETURN NEW; END $$ LANGUAGE plpgsql; ``` **Reasoning:** - Native PostgreSQL feature (no external service) - Fast with GIN index - Weighted fields (names more important than dates) - Simple lexer (no German stemming initially) **Why not Elasticsearch/Meilisearch?** - Overkill for small to mid-sized clubs - Additional infrastructure complexity - PostgreSQL full-text sufficient for 10k+ members ### Deviations from Initial Plans #### 1. No Ecto Schemas **Original Plan:** Traditional Phoenix with Ecto **Actual:** Ash Resources with AshPostgres **Why:** Ash provides more features with less code (policies, admin, code generation) #### 2. Bidirectional Email Sync **Original Plan:** Single email, always linked **Actual:** Optional link with conditional sync **Why:** Members can exist without user accounts (flexibility requirement) #### 3. UUIDv7 for Members **Original Plan:** Standard UUIDv4 **Actual:** UUIDv7 for members, v4 for others ```elixir # Member uses UUIDv7 (sortable by creation time) uuid_v7_primary_key :id # Users use standard UUID uuid_primary_key :id ``` **Why:** Better database performance, chronological ordering #### 4. No Default Create Action for Users **Decision:** Intentionally exclude default `:create` action ```elixir actions do # Explicitly NO default :create defaults [:read, :destroy] # Use specific create actions instead create :create_user create :register_with_password create :register_with_rauthy end ``` **Why:** Bypass email sync if default create used (safety measure) --- ## Build and Deployment ### Development Workflow **For current setup instructions, see [`README.md`](../README.md#-quick-start-development).** **Key workflow decisions:** - **Just** as task runner: Simplifies common tasks, better than raw mix commands - **Docker Compose** for services: Consistent environments, easy local OIDC testing - **Seed data included**: Realistic test data for development #### Database Migrations **Key migrations in chronological order:** 1. `20250528163901_initial_migration.exs` - Core tables (members, properties, property_types) 2. `20250617090641_member_fields.exs` - Member attributes expansion 3. `20250620110850_add_accounts_domain.exs` - Users & tokens tables 4. `20250912085235_AddSearchVectorToMembers.exs` - Full-text search (tsvector + GIN index) 5. `20250926164519_member_relation.exs` - User-Member link (optional 1:1) 6. `20251016130855_add_constraints_for_user_member_and_property.exs` - Email sync constraints **Learning:** Ash's code generation from resources ensures schema always matches code. #### Environment Variables & Secrets **Key environment variables:** - `SECRET_KEY_BASE` - Phoenix session encryption - `TOKEN_SIGNING_SECRET` - JWT token signing - `OIDC_CLIENT_SECRET` - Rauthy OAuth2 client secret - `DATABASE_URL` - PostgreSQL connection (production only) **Secret management approach:** - Development: `.env` file (gitignored) - Production: `config/runtime.exs` reads from environment - Generation: `mix phx.gen.secret` **For complete setup, see [`README.md`](../README.md#-configuration) and [`README.md - Testing SSO`](../README.md#-testing-sso-locally).** ### Testing **Key testing decisions:** - **Ecto Sandbox:** Isolated, concurrent tests - **ExUnit:** Built-in testing framework (no external dependencies) - **Test structure:** Mirrors application structure (accounts/, membership/, mv_web/) **Important test patterns:** - Email sync edge cases (see `test/accounts/email_sync_edge_cases_test.exs`) - User-Member relationship tests (see `test/accounts/user_member_relationship_test.exs`) - LiveView integration tests **For testing guidelines, see [`CODE_GUIDELINES.md - Testing Standards`](../CODE_GUIDELINES.md#4-testing-standards).** ### Code Quality **Tools in use:** - **Credo** `~> 1.7`: Static code analysis - **Sobelow** `~> 0.14`: Security analysis - **mix_audit** `~> 2.1`: Dependency vulnerability scanning - **mix format**: Auto-formatting (2-space indentation, 120 char line length) **CI/CD:** Drone CI runs linting, formatting checks, tests, and security scans on every push. **Build Status:** [![Build Status](https://drone.dev.local-it.cloud/api/badges/local-it/mitgliederverwaltung/status.svg)](https://drone.dev.local-it.cloud/local-it/mitgliederverwaltung) **For detailed guidelines, see [`CODE_GUIDELINES.md`](../CODE_GUIDELINES.md).** ### Docker Deployment **Deployment strategy:** - **Multi-stage build:** Builder stage (Debian + Elixir) → Runtime stage (Debian slim) - **Assets:** Compiled during build with `mix assets.deploy` - **Releases:** Mix release for production (smaller image, faster startup) - **Migrations:** Run via `Mv.Release.migrate` module **Key decisions:** - **Bandit** instead of Cowboy: Better LiveView performance - **Postgres 16** in production: Stable, well-tested - **Separate dev/prod compose files:** Different needs (dev has Rauthy, Mailcrab) - **Release module** (`Mv.Release`): Handles migrations and seeding in production **For complete deployment instructions, see [`README.md - Production Deployment`](../README.md#-production-deployment).** ### Automated Dependency Updates **Tool:** Renovate (via Drone CI) **Configuration:** `renovate_backend_config.js` **Key decisions:** - **Schedule:** First week of each month (reduces PR noise) - **Grouping:** Mix dependencies, asdf tools, postgres updates grouped - **Disabled:** Elixir/Erlang auto-updates (manual version management via asdf) **Why disabled for Elixir/Erlang?** - OTP version coupling requires careful testing - Version compatibility with dependencies - Manual control preferred for core runtime **For details, see [`CODE_GUIDELINES.md - Dependency Management`](../CODE_GUIDELINES.md#14-dependency-management).** --- ## Testing Strategy ### Test Coverage Areas #### 1. Unit Tests (Domain Logic) **Example: Member Email Validation** ```elixir defmodule Mv.Membership.MemberTest do use Mv.DataCase, async: true describe "email validation" do test "accepts valid email" do assert {:ok, member} = create_member(%{email: "valid@example.com"}) end test "rejects invalid email" do assert {:error, _} = create_member(%{email: "invalid"}) end end end ``` #### 2. Integration Tests (Cross-Domain) **Example: User-Member Relationship** ```elixir defmodule Mv.Accounts.UserMemberRelationshipTest do use Mv.DataCase, async: true test "linking user to member syncs emails" do {:ok, user} = create_user(%{email: "user@example.com"}) {:ok, member} = create_member(%{email: "member@example.com"}) # Link user to member {:ok, updated_member} = link_user_to_member(user, member) # Member email should match user email assert updated_member.email == "user@example.com" end end ``` #### 3. LiveView Tests **Example: Member List Sorting** ```elixir defmodule MvWeb.MemberLive.IndexTest do use MvWeb.ConnCase, async: true import Phoenix.LiveViewTest test "sorting members by last name", %{conn: conn} do {:ok, view, _html} = live(conn, ~p"/members") # Click sort header view |> element("th[phx-click='sort']") |> render_click() # Verify sorted order in view assert has_element?(view, "#member-1") end end ``` #### 4. Component Tests **Example: Search Bar** ```elixir defmodule MvWeb.Components.SearchBarTest do use MvWeb.ConnCase, async: true import Phoenix.LiveViewTest test "renders search input" do assigns = %{search_query: "", id: "search"} html = render_component(&search_bar/1, assigns) assert html =~ "input" assert html =~ ~s(type="search") end end ``` ### Test Data Management **Seed Data:** - Admin user: `admin@mv.local` / `testpassword` - Sample members: Hans Müller, Greta Schmidt, Friedrich Wagner - Linked accounts: Maria Weber, Thomas Klein - Property types: String, Date, Boolean, Email **Test Helpers:** ```elixir # test/support/fixtures.ex def member_fixture(attrs \\ %{}) do default_attrs = %{ first_name: "Test", last_name: "User", email: "test#{System.unique_integer()}@example.com" } {:ok, member} = default_attrs |> Map.merge(attrs) |> Mv.Membership.create_member() member end ``` **Testing best practices applied:** - Async by default with Ecto Sandbox - Descriptive test names explaining behavior - Arrange-Act-Assert pattern - One assertion per test - Fixtures for test data setup **For complete guidelines, see [`CODE_GUIDELINES.md - Testing Standards`](../CODE_GUIDELINES.md#4-testing-standards).** --- ## Common Issues and Solutions ### 1. Email Synchronization Conflicts **Issue:** Creating user/member with email that exists in other table (unlinked). **Error:** ``` "Email already used by another (unlinked) member" ``` **Root Cause:** Custom validation prevents cross-table email conflicts for linked entities. **Solution:** - Link existing entities first - Or use different email - Validation only applies to linked entities **Documentation:** `docs/email-sync.md` ### 2. Ash Migration Conflicts **Issue:** Migrations out of sync with resource definitions. **Symptoms:** - Migration fails - Columns don't match resource attributes - Foreign keys missing **Solution:** ```bash # Rollback conflicting migrations mix ash_postgres.rollback -n 1 # Delete migration files rm priv/repo/migrations/_*.exs rm priv/resource_snapshots/repo/_*.json # Regenerate mix ash.codegen --name # Or use Just helper just regen-migrations ``` ### 3. OIDC Authentication Not Working **Issue:** OIDC login fails with redirect error. **Symptoms:** - "Invalid redirect_uri" - "Client not found" **Checklist:** 1. ✅ Rauthy running: `docker compose ps` 2. ✅ Client created in Rauthy admin panel 3. ✅ Redirect URI matches exactly: `http://localhost:4000/auth/user/rauthy/callback` 4. ✅ OIDC_CLIENT_SECRET in .env 5. ✅ App restarted after .env update **Debug:** ```bash # Check Rauthy logs docker compose logs rauthy # Check app logs for OIDC errors mix phx.server ``` ### 4. Full-Text Search Not Working **Issue:** Search returns no results. **Symptoms:** - Empty search results - tsvector not updated **Solution:** ```sql -- Check if trigger exists SELECT tgname FROM pg_trigger WHERE tgrelid = 'members'::regclass; -- Manually update search_vector (if trigger missing) UPDATE members SET search_vector = setweight(to_tsvector('simple', first_name), 'A') || setweight(to_tsvector('simple', last_name), 'A'); -- Or recreate trigger psql mv_dev < priv/repo/migrations/20250912085235_AddSearchVectorToMembers.exs ``` ### 5. Docker Build Fails **Issue:** Production Docker build fails. **Common causes:** - Mix dependencies compilation errors - Asset compilation fails - Missing environment variables **Solution:** ```bash # Clean build cache docker builder prune # Build with no cache docker build --no-cache -t mila:latest . # Check build logs for specific error docker build -t mila:latest . 2>&1 | tee build.log ``` ### 6. Test Failures After Migration **Issue:** Tests fail after running new migration. **Symptoms:** - `column does not exist` - `relation does not exist` **Solution:** ```bash # Reset test database MIX_ENV=test mix ash.reset # Or manually MIX_ENV=test mix ecto.drop MIX_ENV=test mix ash.setup # Run tests again mix test ``` ### 7. Credo/Formatter Conflicts **Issue:** CI fails with formatting/style issues. **Solution:** ```bash # Format all files mix format # Check what would change mix format --check-formatted --dry-run # Run Credo mix credo --strict # Auto-fix some issues mix credo suggest --format=oneline ``` ### 8. Property Value Type Mismatch **Issue:** Property value doesn't match property_type definition. **Error:** ``` "Expected type :integer, got :string" ``` **Solution:** Ensure property value matches property_type.value_type: ```elixir # Property Type: value_type = :integer property_type = get_property_type("age") # Property Value: must be integer union type {:ok, property} = create_property(%{ value: %{type: :integer, value: 25}, # Not "25" as string property_type_id: property_type.id }) ``` --- ## Future Improvements ### Planned Features (Roadmap) Based on open milestones: https://git.local-it.org/local-it/mitgliederverwaltung/pulls #### High Priority 1. **Roles & Permissions** 🔐 - Admin, Treasurer, Member roles - Resource-level permissions - Ash policies for authorization 2. **Payment Tracking** 💰 - Payment history - Fee calculations - Due dates and reminders - Import from vereinfacht API 3. **Intuitive Navigation** 🧭 - Breadcrumbs - Better menu structure - Search in navigation #### Medium Priority 4. **Email Communication** 📧 - Send emails to members - Email templates - Bulk email (with consent) 5. **Member Self-Service** 👤 - Members update own data - Online application - Profile management 6. **Advanced Filtering** 🔍 - Multi-field filters - Saved filter presets - Export filtered results 7. **Accessibility Improvements** ♿ - WCAG 2.1 AA compliance - Screen reader optimization - Keyboard navigation - High contrast mode #### Low Priority 8. **Document Management** 📄 - Attach files to members - Document templates - Digital signatures 9. **Reporting & Analytics** 📊 - Membership statistics - Payment reports - Custom reports 10. **Staging Environment** 🔧 - Separate staging server - Automated deployments - Preview branches ### Technical Debt 1. **German Stemming for Search** - Current: Simple lexer - Needed: German language support in full-text search - Library: `ts_german` or Snowball 2. **Performance Optimization** - Add more indexes based on query patterns - Optimize N+1 queries (use Ash preloading) - Lazy loading for large datasets 3. **Error Handling Improvements** - Better user-facing error messages - Error tracking (Sentry integration?) - Graceful degradation 4. **Test Coverage** - Current: ~70% (estimated) - Goal: >85% - Focus: Email sync edge cases, validation logic 5. **Documentation** - User manual - Admin guide - API documentation (if needed) - Video tutorials ### Infrastructure Improvements 1. **Monitoring** - Application metrics (Prometheus?) - Error tracking - Performance monitoring 2. **Backup Strategy** - Automated database backups - Point-in-time recovery - Backup testing 3. **Scalability** - Database connection pooling - Caching strategy (ETS, Redis?) - CDN for assets 4. **Security Hardening** - Rate limiting - CSRF protection (already enabled) - Security headers - Regular security audits --- ## Team Knowledge Base ### Key Contacts & Resources **Repository:** https://git.local-it.org/local-it/mitgliederverwaltung **CI/CD:** https://drone.dev.local-it.cloud/local-it/mitgliederverwaltung **Issues:** https://git.local-it.org/local-it/mitgliederverwaltung/-/issues **Pull Requests:** https://git.local-it.org/local-it/mitgliederverwaltung/pulls ### Development Conventions #### Commit Messages Follow conventional commits: ``` :