mitgliederverwaltung/docs/development-progress-log.md

# Development Progress Log

**Project:** Mila - Membership Management System  
**Repository:** https://git.local-it.org/local-it/mitgliederverwaltung  
**License:** AGPLv3  
**Status:** Early Development (⚠️ Not Production Ready)

---

## Table of Contents

1. [Project Overview](#project-overview)
2. [Setup and Foundation](#setup-and-foundation)
3. [Major Features Implementation](#major-features-implementation)
4. [Implementation Decisions](#implementation-decisions)
5. [Build and Deployment](#build-and-deployment)
6. [Testing Strategy](#testing-strategy)
7. [Common Issues and Solutions](#common-issues-and-solutions)
8. [Future Improvements](#future-improvements)
9. [Team Knowledge Base](#team-knowledge-base)

---

## Project Overview

### Vision
Simple, usable, self-hostable membership management for small to mid-sized clubs.

### Philosophy
*"Software should help people spend less time on administration and more time on their community."*

### Core Principles
- ✅ **Simple:** Focused on essential club needs
- ✅ **Usable:** Clean, accessible UI for everyday volunteers
- ✅ **Flexible:** Customizable data fields, role-based permissions
- ✅ **Open:** 100% free and open source, no vendor lock-in
- ✅ **Self-hostable:** Full control over data and deployment

### Target Users
- Small to mid-sized clubs
- Volunteer administrators (non-technical)
- Club members (self-service access)

---

## Setup and Foundation

### Initial Project Setup

For **current setup instructions**, see [`README.md`](../README.md#-quick-start-development).

**Historical context:**

#### 1. Phoenix Project Initialization (Sprint 0)

```bash
mix phx.new mv --no-ecto --no-mailer
```

**Reasoning:**
- `--no-ecto`: Using Ash Framework with AshPostgres instead
- `--no-mailer`: Added Swoosh later for better control

#### 2. Technology Choices

**For complete tech stack details, see [`CODE_GUIDELINES.md`](../CODE_GUIDELINES.md#project-context).**

**Key decisions:**
- **Elixir 1.18.3 + OTP 27**: Latest stable versions for performance
- **Ash Framework 3.0**: Declarative resource layer, reduces boilerplate
- **Phoenix LiveView 1.1**: Real-time UI without JavaScript complexity
- **Tailwind CSS 4.0**: Utility-first styling with custom build
- **PostgreSQL 17**: Advanced features (full-text search, JSONB, citext)
- **Bandit**: Modern HTTP server, better than Cowboy for LiveView

#### 3. Version Management (asdf)

**Tool:** asdf 0.16.5 for consistent environments across team

**Versions pinned in `.tool-versions`:**
- Elixir 1.18.3-otp-27
- Erlang 27.3.4  
- Just 1.43.0

#### 4. Database Setup

**PostgreSQL Extensions:**
```sql
CREATE EXTENSION IF NOT EXISTS "uuid-ossp";  -- UUID generation
CREATE EXTENSION IF NOT EXISTS "citext";      -- Case-insensitive text
```

**Migration Strategy:**
```bash
mix ash.codegen --name <migration_name>  # Generate from Ash resources
mix ash.migrate                           # Apply migrations
```

**Reasoning:** Ash generates migrations from resource definitions, ensuring schema matches code.

#### 5. Development Workflow (Just)

Chose **Just** over Makefile for:
- Better error messages
- Cleaner syntax
- Cross-platform compatibility

**Core commands:** See [README.md](../README.md#-development)

---

## Major Features Implementation

### Sprint History & Key Pull Requests

Based on closed PRs from https://git.local-it.org/local-it/mitgliederverwaltung/pulls?state=closed:

#### Phase 1: Foundation (Sprint 0-2)

**Sprint 0 - Vorarbeit**
- Initial project setup
- Technology stack decisions
- Repository structure

**Sprint 2 - 07.05 - 28.05**
- Basic Phoenix setup
- Initial database schema
- Development environment configuration

#### Phase 2: Core Features (Sprint 3-5)

**Sprint 3 - 28.05 - 09.07**
- Member CRUD operations
- Basic custom field system
- Initial UI with Tailwind CSS

**Sprint 4 - 09.07 - 30.07**
- CustomFieldValue types implementation
- Data validation
- Error handling improvements

**Sprint 5 - 31.07 - 11.09**

**PR #138:** *Customize login screen and members as landing page* (closes #68, #137)
- Custom login UI with DaisyUI
- Members page as default landing
- Improved navigation flow

**PR #139:** *Added PR and issue templates* (closes #129)
- GitHub/GitLab issue templates
- PR template for consistent reviews
- Contribution guidelines

**PR #147:** *Add seed data for members*
- Comprehensive seed data
- Test users and members
- CustomFieldValue type examples

#### Phase 3: Search & Navigation (Sprint 6)

**Sprint 6 - 11.09 - 02.10**

**PR #163:** *Implement full-text search for members* (closes #11) 🔍
- PostgreSQL full-text search with tsvector
- Weighted search fields (names: A, email/notes: B, contact: C)
- GIN index for performance
- Auto-updating trigger
- Migration: `20250912085235_AddSearchVectorToMembers.exs`

```elixir
# Search implementation highlights
attribute :search_vector, AshPostgres.Tsvector,
  writable?: false,
  public?: false,
  select_by_default?: false
```

**Key learnings:**
- Simple lexer used (no German stemming initially)
- Weighted fields improve relevance
- GIN index essential for performance

#### Phase 4: Sorting & User Management (Sprint 7)

**Sprint 7 - 02.10 - 23.10**

**PR #166:** *Sorting header for members list* (closes #152, #175)
- Sortable table headers component
- Multi-column sorting support
- Visual indicators for sort direction
- Accessibility improvements (ARIA labels)

**PR #172:** *Create logical link between users and members* (closes #164)
- Optional 1:1 relationship (0..1 ↔ 0..1)
- User `belongs_to` Member
- Member `has_one` User
- Foundation for email sync feature
- Migration: `20250926164519_member_relation.exs`

**PR #148:** *Fix error when deleting members*
- Cascade delete handling
- Proper foreign key constraints
- Error message improvements

**PR #173:** *Link to user data from profile button* (closes #170)
- Profile navigation improvements
- User-member relationship display
- Better UX for linked accounts

**PR #178:** *Polish README* (closes #158)
- Updated documentation
- Better onboarding instructions
- Screenshots and examples

#### Phase 5: Email Synchronization (Sprint 8)

**Sprint 8 - 23.10 - 13.11**

**PR #181:** *Sync email between user and member* (closes #167) ✉️
- Bidirectional email synchronization between User and Member
- User.email as source of truth on linking
- Custom Ash changes with conditional execution
- Complex validation logic to prevent conflicts
- Migration: `20251016130855_add_constraints_for_user_member_and_property.exs`

**See:** [`docs/email-sync.md`](email-sync.md) for complete sync rules and decision tree.

---

#### Phase 6: Search Enhancement & OIDC Improvements (Sprint 9)

**Sprint 9 - 01.11 - 13.11 (finalized)**

**PR #187:** *Implement fuzzy search* (closes #162) 🔍
- PostgreSQL `pg_trgm` extension for trigram-based fuzzy search
- 6 new GIN trigram indexes on members table:
  - first_name, last_name, email, city, street, notes
- Combined search strategy: Full-text (tsvector) + Trigram similarity
- Configurable similarity threshold (default 0.2)
- Migration: `20251001141005_add_trigram_to_members.exs`
- 443 lines of comprehensive tests

**Key learnings:**
- Trigram indexes significantly improve fuzzy matching
- Combined FTS + trigram provides best user experience
- word_similarity() better for partial word matching than similarity()
- Similarity threshold of 0.2 balances precision and recall

**Implementation highlights:**
```elixir
# New Ash action: :search with fuzzy matching
read :search do
  argument :query, :string, allow_nil?: true
  argument :similarity_threshold, :float, allow_nil?: true
  # Uses fragment() for pg_trgm operators: %, similarity(), word_similarity()
end

# Public function for LiveView usage
def fuzzy_search(query, opts) do
  Ash.Query.for_read(query, :search, %{query: query_string})
end
```

---

**PR #192:** *OIDC handling and linking* (closes #171) 🔐
- Secure OIDC account linking with password verification
- Security fix: Filter OIDC sign-in by `oidc_id` instead of email
- New custom error: `PasswordVerificationRequired`
- New validation: `OidcEmailCollision` for email conflict detection
- New LiveView: `LinkOidcAccountLive` for interactive linking
- Automatic linking for passwordless users (no password prompt)
- Password verification required for password-protected accounts
- Comprehensive security logging for audit trail
- Locale persistence via secure cookie (1 year TTL)
- Documentation: `docs/oidc-account-linking.md`

**Security improvements:**
- Prevents account takeover via OIDC email matching
- Password verification before linking OIDC to password accounts
- All linking attempts logged with appropriate severity
- CSRF protection on linking forms
- Secure cookie flags: `http_only`, `secure`, `same_site: "Lax"`

**Test coverage:**
- 5 new comprehensive test files (1,793 lines total):
  - `user_authentication_test.exs` (265 lines)
  - `oidc_e2e_flow_test.exs` (415 lines)
  - `oidc_email_update_test.exs` (271 lines)
  - `oidc_password_linking_test.exs` (496 lines)
  - `oidc_passwordless_linking_test.exs` (210 lines)
- Extended `oidc_integration_test.exs` (+136 lines)

**Key learnings:**
- Account linking requires careful security considerations
- Passwordless users should be auto-linked (better UX)
- Audit logging essential for security-critical operations
- Locale persistence improves user experience post-logout

---

**PR #193:** *Docs, Code Guidelines and Progress Log* 📚
- Complete project documentation suite (5,554 lines)
- New documentation files:
  - `CODE_GUIDELINES.md` (2,578 lines) - Comprehensive development guidelines
  - `docs/database-schema-readme.md` (392 lines) - Database documentation
  - `docs/database_schema.dbml` (329 lines) - DBML schema definition
  - `docs/development-progress-log.md` (1,227 lines) - This file
  - `docs/feature-roadmap.md` (743 lines) - Feature planning and roadmap
- Reduced redundancy in README.md (links to detailed docs)
- Cross-referenced documentation for easy navigation

---

**PR #201:** *Code documentation and refactoring* 🔧
- @moduledoc for ALL modules (51 modules documented)
- @doc for all public functions
- Enabled Credo `ModuleDoc` check (enforces documentation standards)
- Refactored complex functions:
  - `MemberLive.Index.handle_event/3` - Split sorting logic into smaller functions
  - `AuthController.handle_auth_failure/2` - Reduced cyclomatic complexity
- Documentation coverage: 100% for core modules

**Key learnings:**
- @moduledoc enforcement improves code maintainability
- Refactoring complex functions improves readability
- Documentation should explain "why" not just "what"
- Credo helps maintain consistent code quality

---

**PR #208:** *Show custom fields per default in member overview* 🔧
- added show_in_overview as attribute to custom fields
- show custom fields in member overview per default
- can be set to false in the settings for the specific custom field

## Implementation Decisions

### Architecture Patterns

#### 1. Ash Framework Over Traditional Phoenix

**Decision:** Use Ash Framework as the primary data layer instead of traditional Ecto contexts.

**Reasoning:**
- **Declarative resource definitions** reduce boilerplate
- **Built-in authorization** with policies
- **Type safety** with calculations and aggregates
- **Code generation** for migrations

**Trade-offs:**
- Steeper learning curve
- Less common in Phoenix community
- Newer ecosystem (fewer resources)
- More opinionated structure

**Outcome:** 
- ✅ Faster feature development
- ✅ Consistent API across resources
- ⚠️ Requires team training

#### 2. Domain-Driven Design

**Decision:** Organize by business domains (Accounts, Membership) rather than technical layers.

**Reasoning:**
- Clear separation of concerns
- Business logic separate from web layer
- Scalable for future domains (payments, communications)

**For detailed project structure, see [`CODE_GUIDELINES.md`](../CODE_GUIDELINES.md#11-project-structure).**

#### 3. Bidirectional Email Sync

**Problem:** Users and Members can exist independently, but when linked, emails must stay synchronized.

**Solution:** Custom Ash changes with conditional execution

**Why not simpler approaches?**
- ❌ Single email table: Too restrictive (members without users need emails)
- ❌ Always sync: Performance concerns, unnecessary for unlinked entities
- ❌ Manual sync: Error-prone, inconsistent
- ✅ Conditional sync with validations: Flexible, safe, performant

**Complete documentation:** See [`docs/email-sync.md`](email-sync.md) for decision tree and sync rules.

#### 4. CustomFieldValue System (EAV Pattern)

**Implementation:** Entity-Attribute-Value pattern with union types

```elixir
# CustomFieldValue Type defines schema
defmodule Mv.Membership.CustomField do
  attribute :name, :string           # "Membership Number"
  attribute :value_type, :atom       # :string, :integer, :boolean, :date, :email
  attribute :immutable, :boolean     # Can't change after creation
  attribute :required, :boolean      # All members must have this
  attribute :show_in_overview, :boolean # "If true, this custom field will be displayed in the member overview table"
end

# CustomFieldValue stores values
defmodule Mv.Membership.CustomFieldValue do
  attribute :value, :union,          # Polymorphic value storage
    constraints: [
      types: [
        string: [type: :string],
        integer: [type: :integer],
        boolean: [type: :boolean],
        date: [type: :date],
        email: [type: Mv.Membership.Email]
      ]
    ]
  belongs_to :member
  belongs_to :custom_field
end
```

**Reasoning:**
- Clubs need different custom fields
- No schema migrations for new fields
- Type safety with union types
- Centralized custom field management

**Constraints:**
- One custom field value per custom field per member (composite unique index)
- Properties deleted with member (CASCADE)
- CustomFieldValue types protected if in use (RESTRICT)

#### 5. Authentication Strategy

**Multi-Strategy Authentication:**

```elixir
authentication do
  strategies do
    # Password-based
    password :password do
      identity_field :email
      hash_provider AshAuthentication.BcryptProvider
    end
    
    # OIDC (Rauthy)
    oidc :rauthy do
      client_id Mv.Secrets
      base_url Mv.Secrets
      client_secret Mv.Secrets
    end
  end
end
```

**Reasoning:**
- Flexibility: Clubs choose authentication method
- Self-hosting: OIDC with Rauthy (open source)
- Fallback: Password auth for non-SSO users
- Security: bcrypt for password hashing

**Token Management:**
- Store all tokens (store_all_tokens? true)
- JWT-based sessions
- Token revocation support

#### 6. UI Framework Choice

**Tailwind CSS + DaisyUI**

**Reasoning:**
- **Tailwind:** Utility-first, no custom CSS
- **DaisyUI:** Pre-built components, consistent design
- **Heroicons:** Icon library, inline SVG
- **Phoenix LiveView:** Server-rendered, minimal JavaScript

**Trade-offs:**
- Larger HTML (utility classes)
- Learning curve for utility-first CSS
- ✅ Faster development
- ✅ Consistent styling
- ✅ Mobile-responsive out of the box

#### 7. Search Implementation (Full-Text + Fuzzy)

**Two-Tiered Search Strategy:**

**A) Full-Text Search (tsvector + GIN Index)**

```sql
-- Auto-updating trigger
CREATE FUNCTION members_search_vector_trigger() RETURNS trigger AS $$
BEGIN
  NEW.search_vector :=
    setweight(to_tsvector('simple', coalesce(NEW.first_name, '')), 'A') ||
    setweight(to_tsvector('simple', coalesce(NEW.last_name, '')), 'A') ||
    setweight(to_tsvector('simple', coalesce(NEW.email, '')), 'B') ||
    setweight(to_tsvector('simple', coalesce(NEW.notes, '')), 'B') ||
    setweight(to_tsvector('simple', coalesce(NEW.city, '')), 'C') ||
    -- ... more fields
  RETURN NEW;
END
$$ LANGUAGE plpgsql;
```

**B) Fuzzy Search (pg_trgm + Trigram GIN Indexes)**

Added November 2025 (PR #187):

```elixir
# Ash action combining FTS + trigram similarity
read :search do
  argument :query, :string
  argument :similarity_threshold, :float

  prepare fn query, _ctx ->
    # 1. Full-text search (tsvector)
    # 2. Trigram similarity (%, similarity(), word_similarity())
    # 3. Substring matching (contains, ilike)
  end
end
```

**6 Trigram Indexes:**
- first_name, last_name, email, city, street, notes
- GIN index with `gin_trgm_ops` operator class

**Reasoning:**
- Native PostgreSQL features (no external service)
- Combined approach handles typos + partial matches
- Fast with GIN indexes
- Simple lexer (no German stemming initially)
- Similarity threshold configurable (default 0.2)

**Why not Elasticsearch/Meilisearch?**
- Overkill for small to mid-sized clubs
- Additional infrastructure complexity
- PostgreSQL full-text + fuzzy sufficient for 10k+ members
- Better integration with existing stack

### Deviations from Initial Plans

#### 1. No Ecto Schemas

**Original Plan:** Traditional Phoenix with Ecto  
**Actual:** Ash Resources with AshPostgres

**Why:** Ash provides more features with less code (policies, admin, code generation)

#### 2. Bidirectional Email Sync

**Original Plan:** Single email, always linked  
**Actual:** Optional link with conditional sync

**Why:** Members can exist without user accounts (flexibility requirement)

#### 3. UUIDv7 for Members

**Original Plan:** Standard UUIDv4  
**Actual:** UUIDv7 for members, v4 for others

```elixir
# Member uses UUIDv7 (sortable by creation time)
uuid_v7_primary_key :id

# Users use standard UUID
uuid_primary_key :id
```

**Why:** Better database performance, chronological ordering

#### 4. No Default Create Action for Users

**Decision:** Intentionally exclude default `:create` action

```elixir
actions do
  # Explicitly NO default :create
  defaults [:read, :destroy]
  
  # Use specific create actions instead
  create :create_user
  create :register_with_password
  create :register_with_rauthy
end
```

**Why:** Bypass email sync if default create used (safety measure)

---

## Build and Deployment

### Development Workflow

**For current setup instructions, see [`README.md`](../README.md#-quick-start-development).**

**Key workflow decisions:**
- **Just** as task runner: Simplifies common tasks, better than raw mix commands
- **Docker Compose** for services: Consistent environments, easy local OIDC testing
- **Seed data included**: Realistic test data for development

#### Database Migrations

**Key migrations in chronological order:**
1. `20250528163901_initial_migration.exs` - Core tables (members, custom_field_values, custom_fields)
2. `20250617090641_member_fields.exs` - Member attributes expansion
3. `20250620110850_add_accounts_domain.exs` - Users & tokens tables
4. `20250912085235_AddSearchVectorToMembers.exs` - Full-text search (tsvector + GIN index)
5. `20250926164519_member_relation.exs` - User-Member link (optional 1:1)
6. `20251001141005_add_trigram_to_members.exs` - Fuzzy search (pg_trgm + 6 GIN trigram indexes)
7. `20251016130855_add_constraints_for_user_member_and_property.exs` - Email sync constraints

**Learning:** Ash's code generation from resources ensures schema always matches code.

#### Environment Variables & Secrets

**Key environment variables:**
- `SECRET_KEY_BASE` - Phoenix session encryption
- `TOKEN_SIGNING_SECRET` - JWT token signing
- `OIDC_CLIENT_SECRET` - Rauthy OAuth2 client secret
- `DATABASE_URL` - PostgreSQL connection (production only)

**Secret management approach:**
- Development: `.env` file (gitignored)
- Production: `config/runtime.exs` reads from environment
- Generation: `mix phx.gen.secret`

**For complete setup, see [`README.md`](../README.md#-configuration) and [`README.md - Testing SSO`](../README.md#-testing-sso-locally).**

### Testing

**Key testing decisions:**
- **Ecto Sandbox:** Isolated, concurrent tests
- **ExUnit:** Built-in testing framework (no external dependencies)
- **Test structure:** Mirrors application structure (accounts/, membership/, mv_web/)

**Important test patterns:**
- Email sync edge cases (see `test/accounts/email_sync_edge_cases_test.exs`)
- User-Member relationship tests (see `test/accounts/user_member_relationship_test.exs`)
- LiveView integration tests

**For testing guidelines, see [`CODE_GUIDELINES.md - Testing Standards`](../CODE_GUIDELINES.md#4-testing-standards).**

### Code Quality

**Tools in use:**
- **Credo** `~> 1.7`: Static code analysis  
- **Sobelow** `~> 0.14`: Security analysis
- **mix_audit** `~> 2.1`: Dependency vulnerability scanning
- **mix format**: Auto-formatting (2-space indentation, 120 char line length)

**CI/CD:** Drone CI runs linting, formatting checks, tests, and security scans on every push.

**Build Status:** [![Build Status](https://drone.dev.local-it.cloud/api/badges/local-it/mitgliederverwaltung/status.svg)](https://drone.dev.local-it.cloud/local-it/mitgliederverwaltung)

**For detailed guidelines, see [`CODE_GUIDELINES.md`](../CODE_GUIDELINES.md).**

### Docker Deployment

**Deployment strategy:**
- **Multi-stage build:** Builder stage (Debian + Elixir) → Runtime stage (Debian slim)
- **Assets:** Compiled during build with `mix assets.deploy`
- **Releases:** Mix release for production (smaller image, faster startup)
- **Migrations:** Run via `Mv.Release.migrate` module

**Key decisions:**
- **Bandit** instead of Cowboy: Better LiveView performance
- **Postgres 16** in production: Stable, well-tested
- **Separate dev/prod compose files:** Different needs (dev has Rauthy, Mailcrab)
- **Release module** (`Mv.Release`): Handles migrations and seeding in production

**For complete deployment instructions, see [`README.md - Production Deployment`](../README.md#-production-deployment).**

### Automated Dependency Updates

**Tool:** Renovate (via Drone CI)

**Configuration:** `renovate_backend_config.js`

**Key decisions:**
- **Schedule:** First week of each month (reduces PR noise)
- **Grouping:** Mix dependencies, asdf tools, postgres updates grouped
- **Disabled:** Elixir/Erlang auto-updates (manual version management via asdf)

**Why disabled for Elixir/Erlang?**
- OTP version coupling requires careful testing
- Version compatibility with dependencies
- Manual control preferred for core runtime

**For details, see [`CODE_GUIDELINES.md - Dependency Management`](../CODE_GUIDELINES.md#14-dependency-management).**

---

## Testing Strategy

### Test Coverage Areas

#### 1. Unit Tests (Domain Logic)

**Example: Member Email Validation**
```elixir
defmodule Mv.Membership.MemberTest do
  use Mv.DataCase, async: true

  describe "email validation" do
    test "accepts valid email" do
      assert {:ok, member} = create_member(%{email: "valid@example.com"})
    end
    
    test "rejects invalid email" do
      assert {:error, _} = create_member(%{email: "invalid"})
    end
  end
end
```

#### 2. Integration Tests (Cross-Domain)

**Example: User-Member Relationship**
```elixir
defmodule Mv.Accounts.UserMemberRelationshipTest do
  use Mv.DataCase, async: true

  test "linking user to member syncs emails" do
    {:ok, user} = create_user(%{email: "user@example.com"})
    {:ok, member} = create_member(%{email: "member@example.com"})
    
    # Link user to member
    {:ok, updated_member} = link_user_to_member(user, member)
    
    # Member email should match user email
    assert updated_member.email == "user@example.com"
  end
end
```

#### 3. LiveView Tests

**Example: Member List Sorting**
```elixir
defmodule MvWeb.MemberLive.IndexTest do
  use MvWeb.ConnCase, async: true
  import Phoenix.LiveViewTest

  test "sorting members by last name", %{conn: conn} do
    {:ok, view, _html} = live(conn, ~p"/members")
    
    # Click sort header
    view
    |> element("th[phx-click='sort']")
    |> render_click()
    
    # Verify sorted order in view
    assert has_element?(view, "#member-1")
  end
end
```

#### 4. Component Tests

**Example: Search Bar**
```elixir
defmodule MvWeb.Components.SearchBarTest do
  use MvWeb.ConnCase, async: true
  import Phoenix.LiveViewTest

  test "renders search input" do
    assigns = %{search_query: "", id: "search"}
    
    html = render_component(&search_bar/1, assigns)
    
    assert html =~ "input"
    assert html =~ ~s(type="search")
  end
end
```

### Test Data Management

**Seed Data:**
- Admin user: `admin@mv.local` / `testpassword`
- Sample members: Hans Müller, Greta Schmidt, Friedrich Wagner
- Linked accounts: Maria Weber, Thomas Klein
- CustomFieldValue types: String, Date, Boolean, Email

**Test Helpers:**
```elixir
# test/support/fixtures.ex
def member_fixture(attrs \\ %{}) do
  default_attrs = %{
    first_name: "Test",
    last_name: "User",
    email: "test#{System.unique_integer()}@example.com"
  }

  {:ok, member} =
    default_attrs
    |> Map.merge(attrs)
    |> Mv.Membership.create_member()

  member
end
```

**Testing best practices applied:**
- Async by default with Ecto Sandbox
- Descriptive test names explaining behavior
- Arrange-Act-Assert pattern
- One assertion per test
- Fixtures for test data setup

**For complete guidelines, see [`CODE_GUIDELINES.md - Testing Standards`](../CODE_GUIDELINES.md#4-testing-standards).**

---

## Common Issues and Solutions

### 1. Email Synchronization Conflicts

**Issue:** Creating user/member with email that exists in other table (unlinked).

**Error:** 
```
"Email already used by another (unlinked) member"
```

**Root Cause:** Custom validation prevents cross-table email conflicts for linked entities.

**Solution:** 
- Link existing entities first
- Or use different email
- Validation only applies to linked entities

**Documentation:** `docs/email-sync.md`

### 2. Ash Migration Conflicts

**Issue:** Migrations out of sync with resource definitions.

**Symptoms:**
- Migration fails
- Columns don't match resource attributes
- Foreign keys missing

**Solution:**
```bash
# Rollback conflicting migrations
mix ash_postgres.rollback -n 1

# Delete migration files
rm priv/repo/migrations/<timestamp>_*.exs
rm priv/resource_snapshots/repo/<resource>_*.json

# Regenerate
mix ash.codegen --name <migration_name>

# Or use Just helper
just regen-migrations <name>
```

### 3. OIDC Authentication Not Working

**Issue:** OIDC login fails with redirect error.

**Symptoms:**
- "Invalid redirect_uri"
- "Client not found"

**Checklist:**
1. ✅ Rauthy running: `docker compose ps`
2. ✅ Client created in Rauthy admin panel
3. ✅ Redirect URI matches exactly: `http://localhost:4000/auth/user/rauthy/callback`
4. ✅ OIDC_CLIENT_SECRET in .env
5. ✅ App restarted after .env update

**Debug:**
```bash
# Check Rauthy logs
docker compose logs rauthy

# Check app logs for OIDC errors
mix phx.server
```

### 4. Full-Text Search Not Working

**Issue:** Search returns no results.

**Symptoms:**
- Empty search results
- tsvector not updated

**Solution:**
```sql
-- Check if trigger exists
SELECT tgname FROM pg_trigger WHERE tgrelid = 'members'::regclass;

-- Manually update search_vector (if trigger missing)
UPDATE members SET search_vector = 
  setweight(to_tsvector('simple', first_name), 'A') ||
  setweight(to_tsvector('simple', last_name), 'A');

-- Or recreate trigger
psql mv_dev < priv/repo/migrations/20250912085235_AddSearchVectorToMembers.exs
```

### 5. Docker Build Fails

**Issue:** Production Docker build fails.

**Common causes:**
- Mix dependencies compilation errors
- Asset compilation fails
- Missing environment variables

**Solution:**
```bash
# Clean build cache
docker builder prune

# Build with no cache
docker build --no-cache -t mila:latest .

# Check build logs for specific error
docker build -t mila:latest . 2>&1 | tee build.log
```

### 6. Test Failures After Migration

**Issue:** Tests fail after running new migration.

**Symptoms:**
- `column does not exist`
- `relation does not exist`

**Solution:**
```bash
# Reset test database
MIX_ENV=test mix ash.reset

# Or manually
MIX_ENV=test mix ecto.drop
MIX_ENV=test mix ash.setup

# Run tests again
mix test
```

### 7. Credo/Formatter Conflicts

**Issue:** CI fails with formatting/style issues.

**Solution:**
```bash
# Format all files
mix format

# Check what would change
mix format --check-formatted --dry-run

# Run Credo
mix credo --strict

# Auto-fix some issues
mix credo suggest --format=oneline
```

### 8. CustomFieldValue Value Type Mismatch

**Issue:** CustomFieldValue value doesn't match custom_field definition.

**Error:**
```
"Expected type :integer, got :string"
```

**Solution:**
Ensure custom field value matches custom_field.value_type:

```elixir
# CustomFieldValue Type: value_type = :integer
custom_field = get_custom_field("age")

# CustomFieldValue Value: must be integer union type
{:ok, custom_field_value} = create_custom_field_value(%{
  value: %{type: :integer, value: 25},  # Not "25" as string
  custom_field_id: custom_field.id
})
```

---

## Future Improvements

### Planned Features (Roadmap)

Based on open milestones: https://git.local-it.org/local-it/mitgliederverwaltung/pulls

#### High Priority

1. **Roles & Permissions** 🔐
   - Admin, Treasurer, Member roles
   - Resource-level permissions
   - Ash policies for authorization

2. **Payment Tracking** 💰
   - Payment history
   - Fee calculations
   - Due dates and reminders
   - Import from vereinfacht API

3. **Intuitive Navigation** 🧭
   - Breadcrumbs
   - Better menu structure
   - Search in navigation

#### Medium Priority

4. **Email Communication** 📧
   - Send emails to members
   - Email templates
   - Bulk email (with consent)

5. **Member Self-Service** 👤
   - Members update own data
   - Online application
   - Profile management

6. **Advanced Filtering** 🔍
   - Multi-field filters
   - Saved filter presets
   - Export filtered results

7. **Accessibility Improvements** ♿
   - WCAG 2.1 AA compliance
   - Screen reader optimization
   - Keyboard navigation
   - High contrast mode

#### Low Priority

8. **Document Management** 📄
   - Attach files to members
   - Document templates
   - Digital signatures

9. **Reporting & Analytics** 📊
   - Membership statistics
   - Payment reports
   - Custom reports

10. **Staging Environment** 🔧
    - Separate staging server
    - Automated deployments
    - Preview branches

### Technical Debt

1. **German Stemming for Search**
   - Current: Simple lexer
   - Needed: German language support in full-text search
   - Library: `ts_german` or Snowball

2. **Performance Optimization**
   - Add more indexes based on query patterns
   - Optimize N+1 queries (use Ash preloading)
   - Lazy loading for large datasets

3. **Error Handling Improvements**
   - Better user-facing error messages
   - Error tracking (Sentry integration?)
   - Graceful degradation

4. **Test Coverage**
   - Current: ~70% (estimated)
   - Goal: >85%
   - Focus: Email sync edge cases, validation logic

5. **Documentation**
   - User manual
   - Admin guide
   - API documentation (if needed)
   - Video tutorials

### Infrastructure Improvements

1. **Monitoring**
   - Application metrics (Prometheus?)
   - Error tracking
   - Performance monitoring

2. **Backup Strategy**
   - Automated database backups
   - Point-in-time recovery
   - Backup testing

3. **Scalability**
   - Database connection pooling
   - Caching strategy (ETS, Redis?)
   - CDN for assets

4. **Security Hardening**
   - Rate limiting
   - CSRF protection (already enabled)
   - Security headers
   - Regular security audits

---

## Team Knowledge Base

### Key Contacts & Resources

**Repository:** https://git.local-it.org/local-it/mitgliederverwaltung  
**CI/CD:** https://drone.dev.local-it.cloud/local-it/mitgliederverwaltung  
**Issues:** https://git.local-it.org/local-it/mitgliederverwaltung/-/issues  
**Pull Requests:** https://git.local-it.org/local-it/mitgliederverwaltung/pulls

### Development Conventions

#### Commit Messages

Follow conventional commits:
```
<type>: <subject>

<body>

<footer>
```

**Types:**
- `feat:` New feature
- `fix:` Bug fix
- `docs:` Documentation
- `style:` Formatting
- `refactor:` Code refactoring
- `test:` Tests
- `chore:` Maintenance

**Example:**
```
feat: add email synchronization for user-member links

Implement bidirectional email sync between users and members.
User.email is source of truth on initial link, then syncs both ways.

Closes #167
```

#### Branch Naming

```
feature/<issue-number>-<short-description>
bugfix/<issue-number>-<short-description>
chore/<description>
```

**Examples:**
- `feature/167-email-sync`
- `bugfix/152-sorting-header`
- `chore/update-dependencies`

#### Pull Request Template

```markdown
## Description
[Brief description of changes]

## Related Issue
Closes #[issue number]

## Type of Change
- [ ] Bug fix
- [ ] New feature
- [ ] Breaking change
- [ ] Documentation update

## Testing
- [ ] Tests pass locally
- [ ] New tests added
- [ ] Manual testing completed

## Checklist
- [ ] Code follows style guidelines
- [ ] Self-review completed
- [ ] Documentation updated
- [ ] No new warnings
```

### Useful Commands Cheat Sheet

```bash
# Development
just run                    # Start everything
just test                   # Run tests
just lint                   # Check code quality
just audit                  # Security checks
just format                 # Format code

# Database
just reset-database         # Reset DB
just seed-database          # Load seeds
just regen-migrations NAME  # Regenerate migrations

# Ash Framework
mix ash.codegen --name NAME          # Generate migration
mix ash.setup                        # Setup database
mix ash.reset                        # Reset database
mix ash_postgres.rollback -n N       # Rollback N migrations

# Testing
mix test                             # All tests
mix test --cover                     # With coverage
mix test test/path/to/test.exs       # Specific file
mix test test/path/to/test.exs:42    # Specific line

# Code Quality
mix format                           # Format all
mix format --check-formatted         # Check format
mix credo                            # Linting
mix credo --strict                   # Strict mode
mix sobelow --config                 # Security scan
mix deps.audit                       # Dependency audit

# Docker
docker compose up -d                 # Start services
docker compose logs -f app           # Follow logs
docker compose down                  # Stop services
docker compose exec app bash         # Shell into container

# Production
docker compose -f docker-compose.prod.yml up -d
docker compose -f docker-compose.prod.yml exec app /app/bin/mv eval "Mv.Release.migrate"
```

### Debugging Tips

**1. IEx (Interactive Elixir):**
```bash
# Start with IEx
iex -S mix phx.server

# In IEx:
iex> import Ecto.Query
iex> alias Mv.Repo
iex> Repo.all(Mv.Membership.Member) |> IO.inspect()
```

**2. LiveView Debugging:**
```elixir
# In LiveView module
def handle_event("some_event", params, socket) do
  IO.inspect(params, label: "DEBUG PARAMS")
  IO.inspect(socket.assigns, label: "DEBUG ASSIGNS")
  # ...
end
```

**3. Query Debugging:**
```elixir
# Enable query logging
config :logger, level: :debug

# In code
import Ecto.Query
Mv.Repo.all(from m in Mv.Membership.Member, where: m.paid == true)
|> IO.inspect(label: "QUERY RESULT")
```

**4. Ash Debugging:**
```elixir
# Enable Ash debug logging
config :ash, :log_level, :debug

# Inspect changeset errors
case Mv.Membership.create_member(attrs) do
  {:error, error} -> IO.inspect(error, label: "ASH ERROR")
  result -> result
end
```

### Common Gotchas

1. **Ash Actions Must Be Defined**
   - Can't use Ecto directly on Ash resources
   - Always use Ash actions: `Ash.create`, `Ash.update`, etc.

2. **Email Sync Only for Linked Entities**
   - Unlinked users/members don't validate cross-table emails
   - Validation kicks in only when linking

3. **Migrations Must Be Run in Order**
   - Ash migrations depend on resource snapshots
   - Don't skip migrations

4. **LiveView Assigns Are Immutable**
   - Must return new socket: `{:noreply, assign(socket, key: value)}`
   - Can't mutate: `socket.assigns.key = value` ❌

5. **Test Database Must Be Reset After Schema Changes**
   - `MIX_ENV=test mix ash.reset` after migrations

6. **Docker Compose Networks**
   - Dev uses `network_mode: host` for Rauthy access
   - Prod should use proper Docker networks

7. **Secrets in runtime.exs, Not config.exs**
   - `config.exs` is compile-time
   - `runtime.exs` is runtime (for env vars)

### Learning Resources

**Ash Framework:**
- Official Docs: https://hexdocs.pm/ash/
- AshPostgres: https://hexdocs.pm/ash_postgres/
- AshAuthentication: https://hexdocs.pm/ash_authentication/

**Phoenix Framework:**
- Phoenix Guides: https://hexdocs.pm/phoenix/overview.html
- LiveView Docs: https://hexdocs.pm/phoenix_live_view/

**Tailwind CSS:**
- Tailwind Docs: https://tailwindcss.com/docs
- DaisyUI Components: https://daisyui.com/components/

**PostgreSQL:**
- Full-Text Search: https://www.postgresql.org/docs/current/textsearch.html
- Constraints: https://www.postgresql.org/docs/current/ddl-constraints.html

---

## Session: User-Member Linking UI Enhancement (2025-01-13)

### Feature Summary
Implemented user-member linking functionality in User Edit/Create views with fuzzy search autocomplete, email conflict handling, and accessibility support.

**Key Features:**
- Autocomplete dropdown with PostgreSQL Trigram fuzzy search
- Keyboard navigation (Arrow keys, Enter, Escape)
- Link/unlink members to user accounts
- Email synchronization between linked entities
- WCAG 2.1 AA compliant (ARIA labels, keyboard accessibility)
- Bilingual UI (English/German)

### Technical Decisions

**1. Search Priority Logic**
Search query takes precedence over email filtering to provide better UX:
- User types → fuzzy search across all unlinked members
- Email matching only used for post-filtering when no search query present

**2. JavaScript Hook for Input Value**
Used minimal JavaScript (~6 lines) for reliable input field updates:
```javascript
// assets/js/app.js
window.addEventListener("phx:set-input-value", (e) => {
  document.getElementById(e.detail.id).value = e.detail.value
})
```
**Rationale:** LiveView DOM patching has race conditions with rapid state changes in autocomplete components. Direct DOM manipulation via `push_event` is the idiomatic LiveView solution for this edge case.

**3. Keyboard Navigation: Hybrid Approach**
Implemented keyboard accessibility with **mostly Server-Side + minimal Client-Side**:

```elixir
# Server-Side: Navigation and Selection (~45 lines)
def handle_event("member_dropdown_keydown", %{"key" => "ArrowDown"}, socket) do
  # Focus management on server
  new_index = min(current + 1, max_index)
  {:noreply, assign(socket, focused_member_index: new_index)}
end
```

```javascript
// Client-Side: Only preventDefault for Enter in forms (~13 lines)
Hooks.ComboBox = {
  mounted() {
    this.el.addEventListener("keydown", (e) => {
      const isDropdownOpen = this.el.getAttribute("aria-expanded") === "true"
      if (e.key === "Enter" && isDropdownOpen) {
        e.preventDefault()  // Prevent form submission
      }
    })
  }
}
```

**Rationale:** 
- Server-Side handles all navigation logic → simpler, testable, follows LiveView best practices
- Client-Side only prevents browser default behavior (form submit on Enter)
- Latency (~20-50ms) is imperceptible for keyboard events without DB queries
- Follows CODE_GUIDELINES "Minimal JavaScript Philosophy"

**Alternative Considered:** Full Client-Side with JavaScript Hook (~80 lines)
- ❌ More complex code
- ❌ State synchronization between client/server
- ✅ Zero latency (but not noticeable in practice)
- **Decision:** Server-Side approach is simpler and sufficient

**4. Fuzzy Search Implementation**
Combined PostgreSQL Full-Text Search + Trigram for optimal results:
```sql
-- FTS for exact word matching
search_vector @@ websearch_to_tsquery('simple', 'greta')
-- Trigram for typo tolerance
word_similarity('gre', first_name) > 0.2
-- Substring for email/IDs
email ILIKE '%greta%'
```

### Key Learnings

#### 1. Ash `manage_relationship` Internals
**Critical Discovery:** During validation, relationship data lives in `changeset.relationships`, NOT `changeset.attributes`:

```elixir
# During validation (manage_relationship processing):
changeset.relationships.member = [{[%{id: "uuid"}], opts}]
changeset.attributes.member_id = nil  # Still nil!

# After action completes:
changeset.attributes.member_id = "uuid"  # Now set
```

**Solution:** Extract member_id from both sources:
```elixir
defp get_member_id_from_changeset(changeset) do
  case Map.get(changeset.relationships, :member) do
    [{[%{id: id}], _opts}] -> id           # New link
    _ -> Ash.Changeset.get_attribute(changeset, :member_id)  # Existing
  end
end
```

**Impact:** Fixed email validation false positives when linking user+member with identical emails.

#### 2. LiveView + JavaScript Integration Patterns

**When to use JavaScript:**
- ✅ Direct DOM manipulation (autocomplete, input values)
- ✅ Browser APIs (clipboard, geolocation)
- ✅ Third-party libraries
- ✅ Preventing browser default behaviors (form submit, scroll)

**When NOT to use JavaScript:**
- ❌ Form submissions
- ❌ Simple show/hide logic
- ❌ Server-side data fetching
- ❌ Keyboard navigation logic (can be done server-side efficiently)

**Pattern:**
```elixir
socket |> push_event("event-name", %{key: value})
```
```javascript
window.addEventListener("phx:event-name", (e) => { /* handle */ })
```

**Keyboard Events Pattern:**
For keyboard navigation in forms, use hybrid approach:
- Server handles navigation logic via `phx-window-keydown`
- Minimal hook only for `preventDefault()` to avoid form submit conflicts
- Result: ~13 lines JS vs ~80 lines for full client-side solution

#### 3. PostgreSQL Trigram Search
Requires `pg_trgm` extension with GIN indexes:
```sql
CREATE INDEX members_first_name_trgm_idx 
  ON members USING GIN(first_name gin_trgm_ops);
```
Supports:
- Typo tolerance: "Gret" finds "Greta"
- Partial matching: "Mit" finds "Mitglied"  
- Substring: "exam" finds "example.com"

#### 4. Server-Side Keyboard Navigation Performance
**Challenge:** Concern that server-side keyboard events would feel laggy.

**Reality Check:**
- LiveView roundtrip: ~20-50ms on decent connection
- Human perception threshold: ~100ms
- Result: **Feels instant** in practice

**Why it works:**
```elixir
# Event handler only updates index (no DB queries)
def handle_event("member_dropdown_keydown", %{"key" => "ArrowDown"}, socket) do
  new_index = min(socket.assigns.focused_member_index + 1, max_index)
  {:noreply, assign(socket, focused_member_index: new_index)}
end
```
- No database queries
- No complex computations
- Just state updates → extremely fast

**When to use Client-Side instead:**
- Complex animations (Canvas, WebGL)
- Real-time gaming
- Continuous interactions (drag & drop, drawing)

**Lesson:** Don't prematurely optimize for latency. Server-side is simpler and often sufficient.

#### 5. Test-Driven Development for Bug Fixes
Effective workflow:
1. Write test that reproduces bug (should fail)
2. Implement minimal fix
3. Verify test passes
4. Refactor while green

**Result:** 355 tests passing, 100% backend coverage for new features.

### Files Changed

**Backend:**
- `lib/membership/member.ex` - `:available_for_linking` action with fuzzy search
- `lib/mv/accounts/user/validations/email_not_used_by_other_member.ex` - Relationship change extraction
- `lib/mv_web/live/user_live/form.ex` - Event handlers, state management

**Frontend:**
- `assets/js/app.js` - Input value hook (6 lines) + ComboBox hook (13 lines)
- `lib/mv_web/live/user_live/form.ex` - Keyboard event handlers, focus management
- `priv/gettext/**/*.po` - 10 new translation keys (DE/EN)

**Tests (NEW):**
- `test/membership/member_fuzzy_search_linking_test.exs`
- `test/accounts/user_member_linking_email_test.exs`
- `test/mv_web/user_live/form_member_linking_ui_test.exs`

### Deployment Notes
- **Assets:** Requires `cd assets && npm run build`
- **Database:** No migrations (uses existing indexes)
- **Config:** No changes required

---

## Conclusion

This project demonstrates a modern Phoenix application built with:
- ✅ **Ash Framework** for declarative resources and policies
- ✅ **Phoenix LiveView** for real-time, server-rendered UI
- ✅ **Tailwind CSS + DaisyUI** for rapid UI development
- ✅ **PostgreSQL** with advanced features (full-text search, UUIDv7)
- ✅ **Multi-strategy authentication** (Password + OIDC)
- ✅ **Complex business logic** (bidirectional email sync)
- ✅ **Flexible data model** (EAV pattern with union types)

**Key Achievements:**
- 🎯 8 sprints completed
- 🚀 82 pull requests merged
- ✅ Core features implemented (CRUD, search, auth, sync)
- 📚 Comprehensive documentation
- 🔒 Security-focused (audits, validations, policies)
- 🐳 Docker-ready for self-hosting

**Next Steps:**
- Implement roles & permissions
- Add payment tracking
- ✅ ~~Improve accessibility (WCAG 2.1 AA)~~ - Keyboard navigation implemented
- Member self-service portal
- Email communication features

---

**Document Version:** 1.2  
**Last Updated:** 2025-11-27  
**Maintainer:** Development Team  
**Status:** Living Document (update as project evolves)