diff --git a/docs/development-progress-log.md b/docs/development-progress-log.md index 04c35f3..2ffe165 100644 --- a/docs/development-progress-log.md +++ b/docs/development-progress-log.md @@ -1,1931 +1,132 @@ # Development Progress Log -**Project:** Mila - Membership Management System -**Repository:** https://git.local-it.org/local-it/mitgliederverwaltung -**License:** AGPLv3 +**Project:** Mila — Membership Management System (AGPLv3) **Status:** Early Development (⚠️ Not Production Ready) ---- - -## Table of Contents - -1. [Project Overview](#project-overview) -2. [Setup and Foundation](#setup-and-foundation) -3. [Major Features Implementation](#major-features-implementation) -4. [Implementation Decisions](#implementation-decisions) -5. [Build and Deployment](#build-and-deployment) -6. [Testing Strategy](#testing-strategy) -7. [Common Issues and Solutions](#common-issues-and-solutions) -8. [Future Improvements](#future-improvements) -9. [Team Knowledge Base](#team-knowledge-base) +A coarse history of what was built and, more importantly, *why*. Per-feature status lives in [`feature-roadmap.md`](feature-roadmap.md); setup/workflow/conventions live in [`README.md`](../README.md) and [`CODE_GUIDELINES.md`](../CODE_GUIDELINES.md). This file keeps the chronological narrative, the architecture decisions and deviations, the migration index, and the hard-won gotchas. --- -## Project Overview +## Chronological narrative -### Vision -Simple, usable, self-hostable membership management for small to mid-sized clubs. +Sprint dates and per-feature ✅/open/missing status are in [`feature-roadmap.md`](feature-roadmap.md); this is the coarse arc. -### Philosophy -*"Software should help people spend less time on administration and more time on their community."* - -### Core Principles -- ✅ **Simple:** Focused on essential club needs -- ✅ **Usable:** Clean, accessible UI for everyday volunteers -- ✅ **Flexible:** Customizable data fields, role-based permissions -- ✅ **Open:** 100% free and open source, no vendor lock-in -- ✅ **Self-hostable:** Full control over data and deployment - -### Target Users -- Small to mid-sized clubs -- Volunteer administrators (non-technical) -- Club members (self-service access) +- **Sprint 0–2 (foundation):** `mix phx.new mv --no-ecto --no-mailer` (Ash instead of Ecto; Swoosh added later). Stack pinned via asdf (`.tool-versions`): Elixir 1.18.3-otp-27, Erlang 27.3.4, Just 1.46.0. PostgreSQL extensions (see `Mv.Repo.installed_extensions/0`): `ash-functions`, `citext`, `pg_trgm`. UUIDv7 comes from a custom `uuid_generate_v7()` SQL function defined in the extensions migration, not from an extension. +- **Sprint 3–5 (core):** Member CRUD, EAV custom-field system, Tailwind/DaisyUI UI, custom login + members landing page, seed data. +- **Sprint 6 (search):** Full-text search — weighted tsvector (names A, email/notes B, contact C), GIN index, auto-updating trigger, simple lexer (no German stemming yet). +- **Sprint 7 (sorting & links):** Sortable table headers (ARIA); optional 1:1 User↔Member link (User `belongs_to` Member, Member `has_one` User) as foundation for email sync. +- **Sprint 8 (email sync):** Bidirectional User↔Member email sync; User.email is source of truth on linking. Rules + decision tree in [`email-sync.md`](email-sync.md). +- **Sprint 9 (search + OIDC):** Fuzzy search (pg_trgm, 6 trigram GIN indexes, combined FTS+trigram, threshold 0.2, `word_similarity`). Secure OIDC account linking with password verification — see [`oidc-account-linking.md`](oidc-account-linking.md). Documentation suite + 100% `@moduledoc` coverage with Credo `ModuleDoc` enforcement. +- **Late 2025 → 2026:** Membership-fee system (types, calendar cycles, settings, status tracking); custom-field search + required-field validation; field-visibility settings; bulk email copy; sidebar navigation (DaisyUI drawer, WCAG 2.1 AA); CSV import (chunked, configurable limits, custom fields); groups (many-to-many, search-integrated); RBAC (4 permission sets, database-backed roles, resource policies, page-permission plug, system-actor pattern); onboarding/join requests (issue #308, TDD); statistics page MVP; SMTP configuration. See [`feature-roadmap.md`](feature-roadmap.md) and the dedicated docs per area. --- -## Setup and Foundation +## Architecture Decisions & Deviations -### Initial Project Setup +### Ash Framework over Ecto contexts +Declarative resources, built-in policy authorization, calculations/aggregates, migration code-gen. Trade-off: steeper learning curve, smaller ecosystem, more opinionated. Original plan was traditional Phoenix + Ecto; switched because Ash delivers more (policies, admin, codegen) with less code. Detailed structure in `CODE_GUIDELINES.md`. -For **current setup instructions**, see [`README.md`](../README.md#-quick-start-development). +### Domain-Driven structure +Organized by business domain (Accounts, Membership, MembershipFees) rather than technical layer, to keep business logic out of the web layer and scale to future domains. -**Historical context:** +### UUIDv7 for Members only +Members use `uuid_v7_primary_key :id` (sortable by creation time → better index locality, chronological ordering); Users and other resources keep standard `uuid_primary_key :id` (v4). The split is deliberate. -#### 1. Phoenix Project Initialization (Sprint 0) +### No default `:create` action on User +`User` defines `defaults [:read]`, declares `:destroy` as a separate explicit named action, and exposes explicit creates (`create_user`, `register_with_password`, `register_with_oidc`) — **never** a default `:create`. Rationale: a default create would bypass the email-sync changes, so it is excluded as a safety measure. -```bash -mix phx.new mv --no-ecto --no-mailer -``` +### Bidirectional email sync — conditional, not always-on +Users and Members can exist independently; emails must stay in sync only when linked. Rejected alternatives: single shared email table (too restrictive — members without users still need emails); always-sync (needless work for unlinked entities); manual sync (error-prone). Chosen: conditional sync via custom Ash changes + validations — flexible, safe, performant. Source of truth is User.email on linking. Full rules + decision tree in [`email-sync.md`](email-sync.md). -**Reasoning:** -- `--no-ecto`: Using Ash Framework with AshPostgres instead -- `--no-mailer`: Added Swoosh later for better control +### Custom fields — EAV with union types +`CustomField` defines the schema (name, `value_type` ∈ {string, integer, boolean, date, email}, required, show_in_overview); `CustomFieldValue` stores the polymorphic value via Ash `:union` type and `belongs_to :member` + `belongs_to :custom_field`. Chosen so clubs can add fields without schema migrations, with type safety. Constraints: one value per (member, custom_field) (composite unique index); values CASCADE-deleted with the member; custom-field types RESTRICT-protected while in use. -#### 2. Technology Choices +### Search — native PostgreSQL, not Elasticsearch/Meilisearch +Two tiers: weighted tsvector full-text (auto-updating trigger) + pg_trgm fuzzy (6 trigram GIN indexes, `similarity`/`word_similarity`, configurable threshold 0.2). Rejected Elasticsearch/Meilisearch: overkill for small/mid clubs, extra infra, and PostgreSQL FTS+fuzzy is sufficient for 10k+ members with better stack integration. German stemming remains a known gap (simple lexer). -**For complete tech stack details, see [`CODE_GUIDELINES.md`](../CODE_GUIDELINES.md#project-context).** +### Authentication — multi-strategy +Password (bcrypt) + OIDC (Rauthy, self-hostable). `store_all_tokens? true`, JWT sessions, token revocation. Clubs pick the method; password is the non-SSO fallback. -**Key decisions:** -- **Elixir 1.18.3 + OTP 27**: Latest stable versions for performance -- **Ash Framework 3.0**: Declarative resource layer, reduces boilerplate -- **Phoenix LiveView 1.1.0-rc.3**: Real-time UI without JavaScript complexity -- **Tailwind CSS 4.0**: Utility-first styling with custom build -- **PostgreSQL 17**: Advanced features (full-text search, JSONB, citext) -- **Bandit**: Modern HTTP server, better than Cowboy for LiveView - -#### 3. Version Management (asdf) - -**Tool:** asdf 0.16.5 for consistent environments across team - -**Versions pinned in `.tool-versions`:** -- Elixir 1.18.3-otp-27 -- Erlang 27.3.4 -- Just 1.46.0 - -#### 4. Database Setup - -**PostgreSQL Extensions:** -```sql -CREATE EXTENSION IF NOT EXISTS "uuid-ossp"; -- UUID generation (via uuid_generate_v7 function) -CREATE EXTENSION IF NOT EXISTS "citext"; -- Case-insensitive text -CREATE EXTENSION IF NOT EXISTS "pg_trgm"; -- Trigram-based fuzzy search -``` - -**Migration Strategy:** -```bash -mix ash.codegen --name # Generate from Ash resources -mix ash.migrate # Apply migrations -``` - -**Reasoning:** Ash generates migrations from resource definitions, ensuring schema matches code. - -#### 5. Development Workflow (Just) - -Chose **Just** over Makefile for: -- Better error messages -- Cleaner syntax -- Cross-platform compatibility - -**Core commands:** See [README.md](../README.md#-development) +### Deployment +Multi-stage Docker (Debian builder → slim runtime), assets via `mix assets.deploy`, Mix release, migrations via `Mv.Release.migrate`. Bandit over Cowboy (LiveView performance). Renovate runs the first week of each month (grouped mix/asdf/postgres); Elixir/Erlang auto-updates are **disabled** — OTP coupling and dependency compatibility require manual asdf control. --- -## Major Features Implementation +## Migration Index (26, chronological) -### Sprint History & Key Pull Requests +Timestamp → intent. Ash generates migrations from resource snapshots, so order matters and they must not be skipped. -Based on closed PRs from https://git.local-it.org/local-it/mitgliederverwaltung/pulls?state=closed: +1. `20250421101957_initialize_extensions_1` — PostgreSQL extensions (ash-functions, citext, pg_trgm) plus the custom `uuid_generate_v7()` SQL function (source of UUIDv7; not an extension) +2. `20250528163901_initial_migration` — core tables (members, custom_field_values, custom_fields — originally property_types/properties) +3. `20250617090641_member_fields` — member attributes expansion +4. `20250617132424_member_delete` — member deletion constraints +5. `20250620110849_add_accounts_domain_extensions` — accounts domain extensions +6. `20250620110850_add_accounts_domain` — users & tokens tables +7. `20250912085235_AddSearchVectorToMembers` — full-text search (tsvector + GIN index) +8. `20250926164519_member_relation` — User–Member link (optional 1:1) +9. `20250926180341_add_unique_email_to_members` — unique email constraint on members +10. `20251001141005_add_trigram_to_members` — fuzzy search (pg_trgm + 6 GIN trigram indexes) +11. `20251016130855_add_constraints_for_user_member_and_property` — email-sync constraints +12. `20251113163600_rename_properties_to_custom_fields_extensions_1` — rename properties extensions +13. `20251113163602_rename_properties_to_custom_fields` — rename property_types → custom_fields, properties → custom_field_values +14. `20251113180429_add_slug_to_custom_fields` — add slug to custom fields +15. `20251113183538_change_custom_field_delete_cascade` — change delete cascade behavior +16. `20251119160509_add_show_in_overview_to_custom_fields` — add show_in_overview flag +17. `20251127134451_add_settings_table` — create settings table (singleton) +18. `20251201115939_add_member_field_visibility_to_settings` — add member_field_visibility JSONB to settings +19. `20251202145404_remove_birth_date_from_members` — remove birth_date field +20. `20251204123714_add_custom_field_values_to_search_vector` — include custom field values in search vector +21. `20251211151449_add_membership_fees_tables` — create membership_fee_types and membership_fee_cycles tables +22. `20251211172549_remove_immutable_from_custom_fields` — remove immutable flag from custom fields +23. `20251211195058_add_membership_fee_settings` — add membership fee settings to settings table +24. `20251218113900_remove_paid_from_members` — remove paid boolean from members (replaced by cycle status) +25. `20260102155350_remove_phone_number_and_make_fields_optional` — remove phone_number; make first_name/last_name optional +26. `20260106161215_add_authorization_domain` — create roles table and add role_id to users -#### Phase 1: Foundation (Sprint 0-2) - -**Sprint 0 - Vorarbeit** -- Initial project setup -- Technology stack decisions -- Repository structure - -**Sprint 2 - 07.05 - 28.05** -- Basic Phoenix setup -- Initial database schema -- Development environment configuration - -#### Phase 2: Core Features (Sprint 3-5) - -**Sprint 3 - 28.05 - 09.07** -- Member CRUD operations -- Basic custom field system -- Initial UI with Tailwind CSS - -**Sprint 4 - 09.07 - 30.07** -- CustomFieldValue types implementation -- Data validation -- Error handling improvements - -**Sprint 5 - 31.07 - 11.09** - -**PR #138:** *Customize login screen and members as landing page* (closes #68, #137) -- Custom login UI with DaisyUI -- Members page as default landing -- Improved navigation flow - -**PR #139:** *Added PR and issue templates* (closes #129) -- GitHub/GitLab issue templates -- PR template for consistent reviews -- Contribution guidelines - -**PR #147:** *Add seed data for members* -- Comprehensive seed data -- Test users and members -- CustomFieldValue type examples - -#### Phase 3: Search & Navigation (Sprint 6) - -**Sprint 6 - 11.09 - 02.10** - -**PR #163:** *Implement full-text search for members* (closes #11) 🔍 -- PostgreSQL full-text search with tsvector -- Weighted search fields (names: A, email/notes: B, contact: C) -- GIN index for performance -- Auto-updating trigger -- Migration: `20250912085235_AddSearchVectorToMembers.exs` - -```elixir -# Search implementation highlights -attribute :search_vector, AshPostgres.Tsvector, - writable?: false, - public?: false, - select_by_default?: false -``` - -**Key learnings:** -- Simple lexer used (no German stemming initially) -- Weighted fields improve relevance -- GIN index essential for performance - -#### Phase 4: Sorting & User Management (Sprint 7) - -**Sprint 7 - 02.10 - 23.10** - -**PR #166:** *Sorting header for members list* (closes #152, #175) -- Sortable table headers component -- Multi-column sorting support -- Visual indicators for sort direction -- Accessibility improvements (ARIA labels) - -**PR #172:** *Create logical link between users and members* (closes #164) -- Optional 1:1 relationship (0..1 ↔ 0..1) -- User `belongs_to` Member -- Member `has_one` User -- Foundation for email sync feature -- Migration: `20250926164519_member_relation.exs` - -**PR #148:** *Fix error when deleting members* -- Cascade delete handling -- Proper foreign key constraints -- Error message improvements - -**PR #173:** *Link to user data from profile button* (closes #170) -- Profile navigation improvements -- User-member relationship display -- Better UX for linked accounts - -**PR #178:** *Polish README* (closes #158) -- Updated documentation -- Better onboarding instructions -- Screenshots and examples - -#### Phase 5: Email Synchronization (Sprint 8) - -**Sprint 8 - 23.10 - 13.11** - -**PR #181:** *Sync email between user and member* (closes #167) ✉️ -- Bidirectional email synchronization between User and Member -- User.email as source of truth on linking -- Custom Ash changes with conditional execution -- Complex validation logic to prevent conflicts -- Migration: `20251016130855_add_constraints_for_user_member_and_property.exs` - -**See:** [`docs/email-sync.md`](email-sync.md) for complete sync rules and decision tree. +(Later migrations exist for join requests `20260309141437_add_join_requests`, join-form settings `20260310114701`, and groups `20260127141620_add_groups_and_member_groups` — added after this index was first cut; extend as needed.) --- -#### Phase 6: Search Enhancement & OIDC Improvements (Sprint 9) +## Gotchas & Hard-won Lessons -**Sprint 9 - 01.11 - 13.11 (finalized)** - -**PR #187:** *Implement fuzzy search* (closes #162) 🔍 -- PostgreSQL `pg_trgm` extension for trigram-based fuzzy search -- 6 new GIN trigram indexes on members table: - - first_name, last_name, email, city, street, notes -- Combined search strategy: Full-text (tsvector) + Trigram similarity -- Configurable similarity threshold (default 0.2) -- Migration: `20251001141005_add_trigram_to_members.exs` -- 443 lines of comprehensive tests - -**Key learnings:** -- Trigram indexes significantly improve fuzzy matching -- Combined FTS + trigram provides best user experience -- word_similarity() better for partial word matching than similarity() -- Similarity threshold of 0.2 balances precision and recall - -**Implementation highlights:** -```elixir -# New Ash action: :search with fuzzy matching -read :search do - argument :query, :string, allow_nil?: true - argument :similarity_threshold, :float, allow_nil?: true - # Uses fragment() for pg_trgm operators: %, similarity(), word_similarity() -end - -# Public function for LiveView usage -def fuzzy_search(query, opts) do - Ash.Query.for_read(query, :search, %{query: query_string}) -end -``` - ---- - -**PR #192:** *OIDC handling and linking* (closes #171) 🔐 -- Secure OIDC account linking with password verification -- Security fix: Filter OIDC sign-in by `oidc_id` instead of email -- New custom error: `PasswordVerificationRequired` -- New validation: `OidcEmailCollision` for email conflict detection -- New LiveView: `LinkOidcAccountLive` for interactive linking -- Automatic linking for passwordless users (no password prompt) -- Password verification required for password-protected accounts -- Comprehensive security logging for audit trail -- Locale persistence via secure cookie (1 year TTL) -- Documentation: `docs/oidc-account-linking.md` - -**Security improvements:** -- Prevents account takeover via OIDC email matching -- Password verification before linking OIDC to password accounts -- All linking attempts logged with appropriate severity -- CSRF protection on linking forms -- Secure cookie flags: `http_only`, `secure`, `same_site: "Lax"` - -**Test coverage:** -- 5 new comprehensive test files (1,793 lines total): - - `user_authentication_test.exs` (265 lines) - - `oidc_e2e_flow_test.exs` (415 lines) - - `oidc_email_update_test.exs` (271 lines) - - `oidc_password_linking_test.exs` (496 lines) - - `oidc_passwordless_linking_test.exs` (210 lines) -- Extended `oidc_integration_test.exs` (+136 lines) - -**Key learnings:** -- Account linking requires careful security considerations -- Passwordless users should be auto-linked (better UX) -- Audit logging essential for security-critical operations -- Locale persistence improves user experience post-logout - ---- - -**PR #193:** *Docs, Code Guidelines and Progress Log* 📚 -- Complete project documentation suite (5,554 lines) -- New documentation files: - - `CODE_GUIDELINES.md` (2,578 lines) - Comprehensive development guidelines - - `docs/database-schema-readme.md` (392 lines) - Database documentation - - `docs/database_schema.dbml` (329 lines) - DBML schema definition - - `docs/development-progress-log.md` (1,227 lines) - This file - - `docs/feature-roadmap.md` (743 lines) - Feature planning and roadmap -- Reduced redundancy in README.md (links to detailed docs) -- Cross-referenced documentation for easy navigation - ---- - -**PR #201:** *Code documentation and refactoring* 🔧 -- @moduledoc for ALL modules (51 modules documented) -- @doc for all public functions -- Enabled Credo `ModuleDoc` check (enforces documentation standards) -- Refactored complex functions: - - `MemberLive.Index.handle_event/3` - Split sorting logic into smaller functions - - `AuthController.handle_auth_failure/2` - Reduced cyclomatic complexity -- Documentation coverage: 100% for core modules - -**Key learnings:** -- @moduledoc enforcement improves code maintainability -- Refactoring complex functions improves readability -- Documentation should explain "why" not just "what" -- Credo helps maintain consistent code quality - ---- - -**PR #208:** *Show custom fields per default in member overview* 🔧 -- added show_in_overview as attribute to custom fields -- show custom fields in member overview per default -- can be set to false in the settings for the specific custom field - -## Implementation Decisions - -### Architecture Patterns - -#### 1. Ash Framework Over Traditional Phoenix - -**Decision:** Use Ash Framework as the primary data layer instead of traditional Ecto contexts. - -**Reasoning:** -- **Declarative resource definitions** reduce boilerplate -- **Built-in authorization** with policies -- **Type safety** with calculations and aggregates -- **Code generation** for migrations - -**Trade-offs:** -- Steeper learning curve -- Less common in Phoenix community -- Newer ecosystem (fewer resources) -- More opinionated structure - -**Outcome:** -- ✅ Faster feature development -- ✅ Consistent API across resources -- ⚠️ Requires team training - -#### 2. Domain-Driven Design - -**Decision:** Organize by business domains (Accounts, Membership) rather than technical layers. - -**Reasoning:** -- Clear separation of concerns -- Business logic separate from web layer -- Scalable for future domains (payments, communications) - -**For detailed project structure, see [`CODE_GUIDELINES.md`](../CODE_GUIDELINES.md#11-project-structure).** - -#### 3. Bidirectional Email Sync - -**Problem:** Users and Members can exist independently, but when linked, emails must stay synchronized. - -**Solution:** Custom Ash changes with conditional execution - -**Why not simpler approaches?** -- ❌ Single email table: Too restrictive (members without users need emails) -- ❌ Always sync: Performance concerns, unnecessary for unlinked entities -- ❌ Manual sync: Error-prone, inconsistent -- ✅ Conditional sync with validations: Flexible, safe, performant - -**Complete documentation:** See [`docs/email-sync.md`](email-sync.md) for decision tree and sync rules. - -#### 4. CustomFieldValue System (EAV Pattern) - -**Implementation:** Entity-Attribute-Value pattern with union types +### Ash `manage_relationship` validation (drove a real bug fix) +During validation (while `manage_relationship` is processing), the linked record lives in **`changeset.relationships`**, not `changeset.attributes` — `member_id` is still `nil` until the action completes: ```elixir -# CustomFieldValue Type defines schema -defmodule Mv.Membership.CustomField do - attribute :name, :string # "Membership Number" - attribute :value_type, :atom # :string, :integer, :boolean, :date, :email - attribute :immutable, :boolean # Can't change after creation - attribute :required, :boolean # All members must have this - attribute :show_in_overview, :boolean # "If true, this custom field will be displayed in the member overview table" -end - -# CustomFieldValue stores values -defmodule Mv.Membership.CustomFieldValue do - attribute :value, :union, # Polymorphic value storage - constraints: [ - types: [ - string: [type: :string], - integer: [type: :integer], - boolean: [type: :boolean], - date: [type: :date], - email: [type: Mv.Membership.Email] - ] - ] - belongs_to :member - belongs_to :custom_field -end -``` - -**Reasoning:** -- Clubs need different custom fields -- No schema migrations for new fields -- Type safety with union types -- Centralized custom field management - -**Constraints:** -- One custom field value per custom field per member (composite unique index) -- Properties deleted with member (CASCADE) -- CustomFieldValue types protected if in use (RESTRICT) - -#### 5. Authentication Strategy - -**Multi-Strategy Authentication:** - -```elixir -authentication do - strategies do - # Password-based - password :password do - identity_field :email - hash_provider AshAuthentication.BcryptProvider - end - - # OIDC (Rauthy) - oidc :rauthy do - client_id Mv.Secrets - base_url Mv.Secrets - client_secret Mv.Secrets - end - end -end -``` - -**Reasoning:** -- Flexibility: Clubs choose authentication method -- Self-hosting: OIDC with Rauthy (open source) -- Fallback: Password auth for non-SSO users -- Security: bcrypt for password hashing - -**Token Management:** -- Store all tokens (store_all_tokens? true) -- JWT-based sessions -- Token revocation support - -#### 6. UI Framework Choice - -**Tailwind CSS + DaisyUI** - -**Reasoning:** -- **Tailwind:** Utility-first, no custom CSS -- **DaisyUI:** Pre-built components, consistent design -- **Heroicons:** Icon library, inline SVG -- **Phoenix LiveView 1.1.0-rc.3:** Server-rendered, minimal JavaScript - -**Trade-offs:** -- Larger HTML (utility classes) -- Learning curve for utility-first CSS -- ✅ Faster development -- ✅ Consistent styling -- ✅ Mobile-responsive out of the box - -#### 7. Search Implementation (Full-Text + Fuzzy) - -**Two-Tiered Search Strategy:** - -**A) Full-Text Search (tsvector + GIN Index)** - -```sql --- Auto-updating trigger -CREATE FUNCTION members_search_vector_trigger() RETURNS trigger AS $$ -BEGIN - NEW.search_vector := - setweight(to_tsvector('simple', coalesce(NEW.first_name, '')), 'A') || - setweight(to_tsvector('simple', coalesce(NEW.last_name, '')), 'A') || - setweight(to_tsvector('simple', coalesce(NEW.email, '')), 'B') || - setweight(to_tsvector('simple', coalesce(NEW.notes, '')), 'B') || - setweight(to_tsvector('simple', coalesce(NEW.city, '')), 'C') || - -- ... more fields - RETURN NEW; -END -$$ LANGUAGE plpgsql; -``` - -**B) Fuzzy Search (pg_trgm + Trigram GIN Indexes)** - -Added November 2025 (PR #187): - -```elixir -# Ash action combining FTS + trigram similarity -read :search do - argument :query, :string - argument :similarity_threshold, :float - - prepare fn query, _ctx -> - # 1. Full-text search (tsvector) - # 2. Trigram similarity (%, similarity(), word_similarity()) - # 3. Substring matching (contains, ilike) - end -end -``` - -**6 Trigram Indexes:** -- first_name, last_name, email, city, street, notes -- GIN index with `gin_trgm_ops` operator class - -**Reasoning:** -- Native PostgreSQL features (no external service) -- Combined approach handles typos + partial matches -- Fast with GIN indexes -- Simple lexer (no German stemming initially) -- Similarity threshold configurable (default 0.2) - -**Why not Elasticsearch/Meilisearch?** -- Overkill for small to mid-sized clubs -- Additional infrastructure complexity -- PostgreSQL full-text + fuzzy sufficient for 10k+ members -- Better integration with existing stack - -### Deviations from Initial Plans - -#### 1. No Ecto Schemas - -**Original Plan:** Traditional Phoenix with Ecto -**Actual:** Ash Resources with AshPostgres - -**Why:** Ash provides more features with less code (policies, admin, code generation) - -#### 2. Bidirectional Email Sync - -**Original Plan:** Single email, always linked -**Actual:** Optional link with conditional sync - -**Why:** Members can exist without user accounts (flexibility requirement) - -#### 3. UUIDv7 for Members - -**Original Plan:** Standard UUIDv4 -**Actual:** UUIDv7 for members, v4 for others - -```elixir -# Member uses UUIDv7 (sortable by creation time) -uuid_v7_primary_key :id - -# Users use standard UUID -uuid_primary_key :id -``` - -**Why:** Better database performance, chronological ordering - -#### 4. No Default Create Action for Users - -**Decision:** Intentionally exclude default `:create` action - -```elixir -actions do - # Explicitly NO default :create - defaults [:read, :destroy] - - # Use specific create actions instead - create :create_user - create :register_with_password - create :register_with_rauthy -end -``` - -**Why:** Bypass email sync if default create used (safety measure) - ---- - -## Build and Deployment - -### Development Workflow - -**For current setup instructions, see [`README.md`](../README.md#-quick-start-development).** - -**Key workflow decisions:** -- **Just** as task runner: Simplifies common tasks, better than raw mix commands -- **Docker Compose** for services: Consistent environments, easy local OIDC testing -- **Seed data included**: Realistic test data for development - -#### Database Migrations - -**Key migrations in chronological order (26 total):** -1. `20250421101957_initialize_extensions_1.exs` - PostgreSQL extensions (uuid-ossp, citext, pg_trgm) -2. `20250528163901_initial_migration.exs` - Core tables (members, custom_field_values, custom_fields - originally property_types/properties) -3. `20250617090641_member_fields.exs` - Member attributes expansion -4. `20250617132424_member_delete.exs` - Member deletion constraints -5. `20250620110849_add_accounts_domain_extensions.exs` - Accounts domain extensions -6. `20250620110850_add_accounts_domain.exs` - Users & tokens tables -7. `20250912085235_AddSearchVectorToMembers.exs` - Full-text search (tsvector + GIN index) -8. `20250926164519_member_relation.exs` - User-Member link (optional 1:1) -9. `20250926180341_add_unique_email_to_members.exs` - Unique email constraint on members -10. `20251001141005_add_trigram_to_members.exs` - Fuzzy search (pg_trgm + 6 GIN trigram indexes) -11. `20251016130855_add_constraints_for_user_member_and_property.exs` - Email sync constraints -12. `20251113163600_rename_properties_to_custom_fields_extensions_1.exs` - Rename properties extensions -13. `20251113163602_rename_properties_to_custom_fields.exs` - Rename property_types → custom_fields, properties → custom_field_values -14. `20251113180429_add_slug_to_custom_fields.exs` - Add slug to custom fields -15. `20251113183538_change_custom_field_delete_cascade.exs` - Change delete cascade behavior -16. `20251119160509_add_show_in_overview_to_custom_fields.exs` - Add show_in_overview flag -17. `20251127134451_add_settings_table.exs` - Create settings table (singleton) -18. `20251201115939_add_member_field_visibility_to_settings.exs` - Add member_field_visibility JSONB to settings -19. `20251202145404_remove_birth_date_from_members.exs` - Remove birth_date field -20. `20251204123714_add_custom_field_values_to_search_vector.exs` - Include custom field values in search vector -21. `20251211151449_add_membership_fees_tables.exs` - Create membership_fee_types and membership_fee_cycles tables -22. `20251211172549_remove_immutable_from_custom_fields.exs` - Remove immutable flag from custom fields -23. `20251211195058_add_membership_fee_settings.exs` - Add membership fee settings to settings table -24. `20251218113900_remove_paid_from_members.exs` - Remove paid boolean from members (replaced by cycle status) -25. `20260102155350_remove_phone_number_and_make_fields_optional.exs` - Remove phone_number, make first_name/last_name optional -26. `20260106161215_add_authorization_domain.exs` - Create roles table and add role_id to users - -**Learning:** Ash's code generation from resources ensures schema always matches code. - -#### Environment Variables & Secrets - -**Key environment variables:** -- `SECRET_KEY_BASE` - Phoenix session encryption -- `TOKEN_SIGNING_SECRET` - JWT token signing -- `OIDC_CLIENT_SECRET` - Rauthy OAuth2 client secret -- `DATABASE_URL` - PostgreSQL connection (production only) - -**Secret management approach:** -- Development: `.env` file (gitignored) -- Production: `config/runtime.exs` reads from environment -- Generation: `mix phx.gen.secret` - -**For complete setup, see [`README.md`](../README.md#-configuration) and [`README.md - Testing SSO`](../README.md#-testing-sso-locally).** - -### Testing - -**Key testing decisions:** -- **Ecto Sandbox:** Isolated, concurrent tests -- **ExUnit:** Built-in testing framework (no external dependencies) -- **Test structure:** Mirrors application structure (accounts/, membership/, mv_web/) - -**Important test patterns:** -- Email sync edge cases (see `test/accounts/email_sync_edge_cases_test.exs`) -- User-Member relationship tests (see `test/accounts/user_member_relationship_test.exs`) -- LiveView integration tests - -**For testing guidelines, see [`CODE_GUIDELINES.md - Testing Standards`](../CODE_GUIDELINES.md#4-testing-standards).** - -### Code Quality - -**Tools in use:** -- **Credo** `~> 1.7`: Static code analysis -- **Sobelow** `~> 0.14`: Security analysis -- **mix_audit** `~> 2.1`: Dependency vulnerability scanning -- **mix format**: Auto-formatting (2-space indentation, 120 char line length) - -**CI/CD:** Drone CI runs linting, formatting checks, tests, and security scans on every push. - -**Build Status:** [![Build Status](https://drone.dev.local-it.cloud/api/badges/local-it/mitgliederverwaltung/status.svg)](https://drone.dev.local-it.cloud/local-it/mitgliederverwaltung) - -**For detailed guidelines, see [`CODE_GUIDELINES.md`](../CODE_GUIDELINES.md).** - -### Docker Deployment - -**Deployment strategy:** -- **Multi-stage build:** Builder stage (Debian + Elixir) → Runtime stage (Debian slim) -- **Assets:** Compiled during build with `mix assets.deploy` -- **Releases:** Mix release for production (smaller image, faster startup) -- **Migrations:** Run via `Mv.Release.migrate` module - -**Key decisions:** -- **Bandit** instead of Cowboy: Better LiveView performance -- **Postgres 16** in production: Stable, well-tested -- **Separate dev/prod compose files:** Different needs (dev has Rauthy, Mailcrab) -- **Release module** (`Mv.Release`): Handles migrations and seeding in production - -**For complete deployment instructions, see [`README.md - Production Deployment`](../README.md#-production-deployment).** - -### Automated Dependency Updates - -**Tool:** Renovate (via Drone CI) - -**Configuration:** `renovate_backend_config.js` - -**Key decisions:** -- **Schedule:** First week of each month (reduces PR noise) -- **Grouping:** Mix dependencies, asdf tools, postgres updates grouped -- **Disabled:** Elixir/Erlang auto-updates (manual version management via asdf) - -**Why disabled for Elixir/Erlang?** -- OTP version coupling requires careful testing -- Version compatibility with dependencies -- Manual control preferred for core runtime - -**For details, see [`CODE_GUIDELINES.md - Dependency Management`](../CODE_GUIDELINES.md#14-dependency-management).** - ---- - -## Testing Strategy - -### Test process environment - -`test/test_helper.exs` clears Vereinfacht and OIDC-related environment variables at startup (same rationale as not hitting real APIs when `.env` is loaded). `Mv.Config` prefers ENV over database settings; without this, OIDC sign-in redirect tests would depend on the developer shell and become flaky. Tests that need specific OIDC env values set them in `setup` and restore with `on_exit`. - -### Test Coverage Areas - -#### 1. Unit Tests (Domain Logic) - -**Example: Member Email Validation** -```elixir -defmodule Mv.Membership.MemberTest do - use Mv.DataCase, async: true - - describe "email validation" do - test "accepts valid email" do - assert {:ok, member} = create_member(%{email: "valid@example.com"}) - end - - test "rejects invalid email" do - assert {:error, _} = create_member(%{email: "invalid"}) - end - end -end -``` - -#### 2. Integration Tests (Cross-Domain) - -**Example: User-Member Relationship** -```elixir -defmodule Mv.Accounts.UserMemberRelationshipTest do - use Mv.DataCase, async: true - - test "linking user to member syncs emails" do - {:ok, user} = create_user(%{email: "user@example.com"}) - {:ok, member} = create_member(%{email: "member@example.com"}) - - # Link user to member - {:ok, updated_member} = link_user_to_member(user, member) - - # Member email should match user email - assert updated_member.email == "user@example.com" - end -end -``` - -#### 3. LiveView Tests - -**Example: Member List Sorting** -```elixir -defmodule MvWeb.MemberLive.IndexTest do - use MvWeb.ConnCase, async: true - import Phoenix.LiveViewTest - - test "sorting members by last name", %{conn: conn} do - {:ok, view, _html} = live(conn, ~p"/members") - - # Click sort header - view - |> element("th[phx-click='sort']") - |> render_click() - - # Verify sorted order in view - assert has_element?(view, "#member-1") - end -end -``` - -#### 4. Component Tests - -**Example: Search Bar** -```elixir -defmodule MvWeb.Components.SearchBarTest do - use MvWeb.ConnCase, async: true - import Phoenix.LiveViewTest - - test "renders search input" do - assigns = %{search_query: "", id: "search"} - - html = render_component(&search_bar/1, assigns) - - assert html =~ "input" - assert html =~ ~s(type="search") - end -end -``` - -### Onboarding / Join (Issue #308, TDD) - -**Subtask 1 – JoinRequest resource and public policies (done):** -- Resource: `Mv.Membership.JoinRequest` with attributes (status, email, first_name, last_name, form_data, schema_version, confirmation_token_hash, confirmation_token_expires_at, submitted_at, etc.), actions `submit` (create), `get_by_confirmation_token_hash` (read), `confirm` (update). Migration: `20260309141437_add_join_requests.exs`. -- Policies: Public actions allowed with `actor: nil` via `Mv.Authorization.Checks.ActorIsNil` (submit, get_by_confirmation_token_hash, confirm); default read remains Forbidden for unauthenticated. -- Domain: `Mv.Membership.submit_join_request/2`, `Mv.Membership.confirm_join_request/2` (token hashing via `JoinRequest.hash_confirmation_token/1`, lookup, expiry check, idempotency for :submitted/:approved/:rejected). -- Test file: `test/membership/join_request_test.exs` – all tests pass; policy test and expired-token test implemented. - -**Subtask 2 – Submit and confirm flow (done):** -- **Unified email layout:** phoenix_swoosh with `MvWeb.EmailLayoutView` (layout) and `MvWeb.EmailsView` (body templates). All transactional emails (join confirmation, user confirmation, password reset) use the same layout. Config: `config :mv, :mail_from, {name, email}` (default `{"Mila", "noreply@example.com"}`); override in runtime.exs. -- **Join confirmation:** Domain wrapper `submit_join_request/2` generates token (or uses optional `:confirmation_token` in attrs for tests), creates JoinRequest via action `:submit`, then sends one email via `MvWeb.Emails.JoinConfirmationEmail`. Route `GET /confirm_join/:token` (JoinConfirmController) updates to `submitted`; idempotent; expired/invalid handled. -- **Senders migrated:** `SendNewUserConfirmationEmail`, `SendPasswordResetEmail` use layout + `Mv.Mailer.mail_from/0`. -- **Cleanup:** Mix task `mix join_requests.cleanup_expired` hard-deletes JoinRequests in `pending_confirmation` with expired `confirmation_token_expires_at` (authorize?: false). For cron/Oban. -- **Gettext:** New email strings in default domain; German translations in de/LC_MESSAGES/default.po; English msgstr filled for email-related strings. -- **PR review follow-ups (Join confirmation):** Join confirmation email uses `Mailer.deliver/2` with `Mailer.smtp_config/0` (same config as test mail). On delivery failure the domain returns `{:error, :email_delivery_failed}` (logged via `Logger.error`), and the JoinLive shows an error message (no success UI). Comment in `submit_join_request/2` clarifies that the raw token is hashed by `JoinRequest.Changes.SetConfirmationToken`. Cleanup task uses `Ash.bulk_destroy` and logs partial errors without halting. Layout uses assigns `app_name` and `locale` (from config/Gettext) instead of hardcoded "Mila" and `lang="de"`. Production `runtime.exs` sets `:mail_from` from ENV (`MAIL_FROM_NAME`, `MAIL_FROM_EMAIL`). Layout reference unified to `"layout.html"`; redundant `put_layout` removed from senders. -- Tests: `join_request_test.exs`, `join_request_submit_email_test.exs`, `join_confirm_controller_test.exs` – all pass. - -**Subtask 3 – Admin: Join form settings (done):** -- **Setting resource** (`lib/membership/setting.ex`): 3 new attributes `join_form_enabled` (boolean, default false), `join_form_field_ids` ({:array, :string} – ordered list of member field names or custom field UUIDs), `join_form_field_required` (:map – field ID → boolean). Added to `:create` and `:update` accept lists. Validation rejects field IDs that are neither valid member field names nor UUID format. Migration: `20260310114701_add_join_form_settings_to_settings.exs`. -- **NormalizeJoinFormSettings** (`lib/membership/setting/changes/normalize_join_form_settings.ex`): Change applied on create/update whenever join form attrs are changing. Ensures email is always in `join_form_field_ids`, forces `join_form_field_required["email"] = true`, drops required flags for fields not in `join_form_field_ids`. -- **Domain** (`lib/membership/membership.ex`): `Mv.Membership.get_join_form_allowlist/0` – returns `[]` when join form is disabled, otherwise a list of `%{id, required, type}` maps (type = `:member_field` or `:custom_field` based on ID format). -- **GlobalSettingsLive** (`lib/mv_web/live/global_settings_live.ex`): New "Join Form" / "Beitrittsformular" section between Club Settings and Vereinfacht. Checkbox to enable/disable, table of selected fields (with Required checkbox per field – email always checked/disabled, other fields can be toggled), "Add field" dropdown with all unselected member fields and custom fields, "Save Join Form Settings" button. State is managed locally (not via AshPhoenix.Form); saved on explicit save click. -- **Translations**: 14 new German strings in `priv/gettext/de/LC_MESSAGES/default.po` (Beitrittsformular, Felder im Beitrittsformular, Feld hinzufügen, etc.). -- Tests: All 13 tests in `test/membership/setting_join_form_test.exs` pass; full test suite 1900 tests, 0 failures. - -### Test Data Management - -**Seed Data:** -- Admin user: `admin@localhost` / `testpassword` (configurable via `ADMIN_EMAIL` env var) -- Sample members: Hans Müller, Greta Schmidt, Friedrich Wagner -- Linked accounts: Maria Weber, Thomas Klein -- CustomFieldValue types: String, Date, Boolean, Email - -**Test Helpers:** -```elixir -# test/support/fixtures.ex -def member_fixture(attrs \\ %{}) do - default_attrs = %{ - first_name: "Test", - last_name: "User", - email: "test#{System.unique_integer()}@example.com" - } - - {:ok, member} = - default_attrs - |> Map.merge(attrs) - |> Mv.Membership.create_member() - - member -end -``` - -**Testing best practices applied:** -- Async by default with Ecto Sandbox -- Descriptive test names explaining behavior -- Arrange-Act-Assert pattern -- One assertion per test -- Fixtures for test data setup - -**For complete guidelines, see [`CODE_GUIDELINES.md - Testing Standards`](../CODE_GUIDELINES.md#4-testing-standards).** - ---- - -## Common Issues and Solutions - -### 1. Email Synchronization Conflicts - -**Issue:** Creating user/member with email that exists in other table (unlinked). - -**Error:** -``` -"Email already used by another (unlinked) member" -``` - -**Root Cause:** Custom validation prevents cross-table email conflicts for linked entities. - -**Solution:** -- Link existing entities first -- Or use different email -- Validation only applies to linked entities - -**Documentation:** `docs/email-sync.md` - -### 2. Ash Migration Conflicts - -**Issue:** Migrations out of sync with resource definitions. - -**Symptoms:** -- Migration fails -- Columns don't match resource attributes -- Foreign keys missing - -**Solution:** -```bash -# Rollback conflicting migrations -mix ash_postgres.rollback -n 1 - -# Delete migration files -rm priv/repo/migrations/_*.exs -rm priv/resource_snapshots/repo/_*.json - -# Regenerate -mix ash.codegen --name - -# Or use Just helper -just regen-migrations -``` - -### 3. OIDC Authentication Not Working - -**Issue:** OIDC login fails with redirect error. - -**Symptoms:** -- "Invalid redirect_uri" -- "Client not found" - -**Checklist:** -1. ✅ Rauthy running: `docker compose ps` -2. ✅ Client created in Rauthy admin panel -3. ✅ Redirect URI matches exactly: `http://localhost:4000/auth/user/oidc/callback` -4. ✅ OIDC_CLIENT_SECRET in .env -5. ✅ App restarted after .env update - -**Debug:** -```bash -# Check Rauthy logs -docker compose logs rauthy - -# Check app logs for OIDC errors -mix phx.server -``` - -### 4. Full-Text Search Not Working - -**Issue:** Search returns no results. - -**Symptoms:** -- Empty search results -- tsvector not updated - -**Solution:** -```sql --- Check if trigger exists -SELECT tgname FROM pg_trigger WHERE tgrelid = 'members'::regclass; - --- Manually update search_vector (if trigger missing) -UPDATE members SET search_vector = - setweight(to_tsvector('simple', first_name), 'A') || - setweight(to_tsvector('simple', last_name), 'A'); - --- Or recreate trigger -psql mv_dev < priv/repo/migrations/20250912085235_AddSearchVectorToMembers.exs -``` - -### 5. Docker Build Fails - -**Issue:** Production Docker build fails. - -**Common causes:** -- Mix dependencies compilation errors -- Asset compilation fails -- Missing environment variables - -**Solution:** -```bash -# Clean build cache -docker builder prune - -# Build with no cache -docker build --no-cache -t mila:latest . - -# Check build logs for specific error -docker build -t mila:latest . 2>&1 | tee build.log -``` - -### 6. Test Failures After Migration - -**Issue:** Tests fail after running new migration. - -**Symptoms:** -- `column does not exist` -- `relation does not exist` - -**Solution:** -```bash -# Reset test database -MIX_ENV=test mix ash.reset - -# Or manually -MIX_ENV=test mix ecto.drop -MIX_ENV=test mix ash.setup - -# Run tests again -mix test -``` - -### 7. Credo/Formatter Conflicts - -**Issue:** CI fails with formatting/style issues. - -**Solution:** -```bash -# Format all files -mix format - -# Check what would change -mix format --check-formatted --dry-run - -# Run Credo -mix credo --strict - -# Auto-fix some issues -mix credo suggest --format=oneline -``` - -### 8. CustomFieldValue Value Type Mismatch - -**Issue:** CustomFieldValue value doesn't match custom_field definition. - -**Error:** -``` -"Expected type :integer, got :string" -``` - -**Solution:** -Ensure custom field value matches custom_field.value_type: - -```elixir -# CustomFieldValue Type: value_type = :integer -custom_field = get_custom_field("age") - -# CustomFieldValue Value: must be integer union type -{:ok, custom_field_value} = create_custom_field_value(%{ - value: %{type: :integer, value: 25}, # Not "25" as string - custom_field_id: custom_field.id -}) -``` - ---- - -## Future Improvements - -### Planned Features (Roadmap) - -Based on open milestones: https://git.local-it.org/local-it/mitgliederverwaltung/pulls - -#### High Priority - -1. **Roles & Permissions** 🔐 - - Admin, Treasurer, Member roles - - Resource-level permissions - - Ash policies for authorization - -2. **Payment Tracking** 💰 - - Payment history - - Fee calculations - - Due dates and reminders - - Import from vereinfacht API - -3. **Intuitive Navigation** 🧭 - - Breadcrumbs - - Better menu structure - - Search in navigation - -#### Medium Priority - -4. **Email Communication** 📧 - - Send emails to members - - Email templates - - Bulk email (with consent) - -5. **Member Self-Service** 👤 - - Members update own data - - Online application - - Profile management - -6. **Advanced Filtering** 🔍 - - Multi-field filters - - Saved filter presets - - Export filtered results - -7. **Accessibility Improvements** ♿ - - WCAG 2.1 AA compliance - - Screen reader optimization - - Keyboard navigation - - High contrast mode - -#### Low Priority - -8. **Document Management** 📄 - - Attach files to members - - Document templates - - Digital signatures - -9. **Reporting & Analytics** 📊 - - Membership statistics - - Payment reports - - Custom reports - -10. **Staging Environment** 🔧 - - Separate staging server - - Automated deployments - - Preview branches - -### Technical Debt - -1. **German Stemming for Search** - - Current: Simple lexer - - Needed: German language support in full-text search - - Library: `ts_german` or Snowball - -2. **Performance Optimization** - - Add more indexes based on query patterns - - Optimize N+1 queries (use Ash preloading) - - Lazy loading for large datasets - -3. **Error Handling Improvements** - - Better user-facing error messages - - Error tracking (Sentry integration?) - - Graceful degradation - -4. **Test Coverage** - - Current: ~70% (estimated) - - Goal: >85% - - Focus: Email sync edge cases, validation logic - -5. **Documentation** - - User manual - - Admin guide - - API documentation (if needed) - - Video tutorials - -### Infrastructure Improvements - -1. **Monitoring** - - Application metrics (Prometheus?) - - Error tracking - - Performance monitoring - -2. **Backup Strategy** - - Automated database backups - - Point-in-time recovery - - Backup testing - -3. **Scalability** - - Database connection pooling - - Caching strategy (ETS, Redis?) - - CDN for assets - -4. **Security Hardening** - - Rate limiting - - CSRF protection (already enabled) - - Security headers - - Regular security audits - ---- - -## Team Knowledge Base - -### Key Contacts & Resources - -**Repository:** https://git.local-it.org/local-it/mitgliederverwaltung -**CI/CD:** https://drone.dev.local-it.cloud/local-it/mitgliederverwaltung -**Issues:** https://git.local-it.org/local-it/mitgliederverwaltung/-/issues -**Pull Requests:** https://git.local-it.org/local-it/mitgliederverwaltung/pulls - -### Development Conventions - -#### Commit Messages - -Follow conventional commits: -``` -: - - - -