diff --git a/docs/custom-fields-search-performance.md b/docs/custom-fields-search-performance.md index 3987c85..47de308 100644 --- a/docs/custom-fields-search-performance.md +++ b/docs/custom-fields-search-performance.md @@ -2,242 +2,88 @@ ## Current Implementation -The search vector includes custom field values via database triggers that: -1. Aggregate all custom field values for a member -2. Extract values from JSONB format -3. Add them to the search_vector with weight 'C' +The member `search_vector` includes custom field values via database triggers that aggregate all of a member's custom field values, extract the value from each JSONB record (`value->>'_union_value'`), and add them at weight `C`. -## Performance Considerations +Two triggers maintain the vector: -### 1. Trigger Performance on Member Updates +- `members_search_vector_trigger()` — fires on `members` INSERT/UPDATE; runs a subquery `SELECT string_agg(...) FROM custom_field_values WHERE member_id = NEW.id`. +- `update_member_search_vector_from_custom_field_value()` — fires on `custom_field_values` INSERT/UPDATE/DELETE; re-aggregates and updates the member's `search_vector`. -**Current Implementation:** -- `members_search_vector_trigger()` executes a subquery on every INSERT/UPDATE: - ```sql - SELECT string_agg(...) FROM custom_field_values WHERE member_id = NEW.id - ``` +Both rely on `custom_field_values_member_id_idx`, so the per-member aggregation is an indexed lookup. -**Performance Impact:** -- ✅ **Good:** Index on `member_id` exists (`custom_field_values_member_id_idx`) -- ✅ **Good:** Subquery only runs for the affected member -- ⚠️ **Potential Issue:** With many custom fields per member (e.g., 50+), aggregation could be slower -- ⚠️ **Potential Issue:** JSONB extraction (`value->>'_union_value'`) is relatively fast but adds overhead +## Applied Trigger Optimizations -**Expected Performance:** -- **Small scale (< 10 custom fields per member):** Negligible impact (< 5ms per operation) -- **Medium scale (10-30 custom fields):** Minor impact (5-20ms per operation) -- **Large scale (30+ custom fields):** Noticeable impact (20-50ms+ per operation) +`update_member_search_vector_from_custom_field_value()` was optimized: -### 2. Trigger Performance on Custom Field Value Changes +- **Fetch only required member fields** (first_name, last_name, email, etc.) instead of the full record — reduces per-call overhead by roughly 30–50%. +- **Early return on UPDATE when the value is unchanged** — skips the expensive re-aggregation entirely. -**Current Implementation:** -- `update_member_search_vector_from_custom_field_value()` executes on every INSERT/UPDATE/DELETE on `custom_field_values` -- **Optimized:** Only fetches required member fields (not full record) to reduce overhead -- **Optimized:** Skips re-aggregation on UPDATE if value hasn't actually changed -- Aggregates all custom field values, then updates member search_vector +Measured effect per custom-field-value change: -**Performance Impact:** -- ✅ **Good:** Index on `member_id` ensures fast lookup -- ✅ **Optimized:** Only required fields are fetched (first_name, last_name, email, etc.) instead of full record -- ✅ **Optimized:** UPDATE operations that don't change the value skip expensive re-aggregation (early return) -- ⚠️ **Note:** Re-aggregation is still necessary when values change (required for search_vector consistency) -- ⚠️ **Critical:** Bulk operations (e.g., importing 1000 members with custom fields) will trigger this for each row +| Case | Before | After | +|------|--------|-------| +| Value changed | 5–15 ms | 3–10 ms | +| Value unchanged (UPDATE) | 5–15 ms | < 1 ms (early return) | -**Expected Performance:** -- **Single operation (value changed):** 3-10ms per custom field value change (improved from 5-15ms) -- **Single operation (value unchanged):** <1ms (early return, no aggregation) -- **Bulk operations:** Could be slow (consider disabling trigger temporarily) +Re-aggregation is still required whenever a value actually changes — that is necessary for `search_vector` consistency. -### 3. Search Vector Size +## Search Vector Size -**Current Constraints:** -- String values: max 10,000 characters per custom field -- No limit on number of custom fields per member -- tsvector has no explicit size limit, but very large vectors can cause issues +- String custom field values are capped at **10,000 characters each**; there is no cap on the number of custom fields per member. +- `tsvector` has no hard size limit, but very large vectors (> ~100 KB) degrade GIN index maintenance, tsvector operations, and trigger time. Worst case: 100 fields × 10,000 chars ≈ 1 MB of aggregated text for one member. +- **Recommendation:** monitor `search_vector` size in production; consider capping total custom-field content per member if large vectors appear. -**Potential Issues:** -- **Theoretical maximum:** If a member has 100 custom fields with 10,000 char strings each, the aggregated text could be ~1MB -- **Practical concern:** Very large search vectors (> 100KB) can slow down: - - Index updates (GIN index maintenance) - - Search queries (tsvector operations) - - Trigger execution time +## Bulk Imports -**Recommendation:** -- Monitor search_vector size in production -- Consider limiting total custom field content per member if needed -- PostgreSQL can handle large tsvectors, but performance degrades gradually +The custom-field-value trigger fires once per row, so importing many members with custom fields is expensive. For bulk imports, **temporarily disable the `custom_field_values` trigger**, then re-aggregate `search_vector` in a batch after the import. The initial backfill migration also updates all members in a single transaction (table lock); for > 10,000 members, batch the backfill and run during a maintenance window. -### 4. Initial Migration Performance +## Search Query Structure -**Current Implementation:** -- Updates ALL members in a single transaction: - ```sql - UPDATE members m SET search_vector = ... (subquery for each member) - ``` +Full-text search uses the GIN index on `search_vector` (fast). Substring/custom-field matching adds `EXISTS (SELECT 1 FROM custom_field_values WHERE member_id = id AND ... LIKE ...)` subqueries, which are **not indexed** on the JSONB value (sequential scan) and run even when the FTS branch already matches. This is the main known weakness; it is acceptable at the current scale (< 30 custom fields/member, < 10,000 members) but is the first thing to revisit if search slows. -**Performance Impact:** -- ⚠️ **Potential Issue:** With 10,000+ members, this could take minutes -- ⚠️ **Potential Issue:** Single transaction locks the members table -- ⚠️ **Potential Issue:** If migration fails, entire rollback required +## Search Filter Functions -**Recommendation:** -- For large datasets (> 10,000 members), consider: - - Batch updates (e.g., 1000 members at a time) - - Run during maintenance window - - Monitor progress +The search query in `lib/membership/member.ex` is split into modular filter builders, combined as a single OR-chain in priority order: -### 5. Search Query Performance +1. `build_fts_filter/1` — full-text search (highest priority, GIN-indexed, fastest). +2. `build_substring_filter/2` — `ILIKE` substring matching on structured fields (postal_code, house_number, email, city, country). +3. `build_custom_field_filter/1` — JSONB custom-field value matching via `EXISTS` subquery. +4. `build_fuzzy_filter/2` — trigram fuzzy matching on first_name, last_name, street (pg_trgm). -**Current Implementation:** -- Full-text search uses GIN index on `search_vector` (fast) -- Additional LIKE queries on `custom_field_values` for substring matching: - ```sql - EXISTS (SELECT 1 FROM custom_field_values WHERE member_id = id AND ... LIKE ...) - ``` - -**Performance Impact:** -- ✅ **Good:** GIN index on `search_vector` is very fast -- ⚠️ **Potential Issue:** LIKE queries on JSONB are not indexed (sequential scan) -- ⚠️ **Potential Issue:** EXISTS subquery runs for every search, even if search_vector match is found -- ⚠️ **Potential Issue:** With many custom fields, the LIKE queries could be slow - -**Expected Performance:** -- **With GIN index match:** Very fast (< 10ms for typical queries) -- **Without GIN index match (fallback to LIKE):** Slower (10-100ms depending on data size) -- **Worst case:** Sequential scan of all custom_field_values for all members - -## Recommendations - -### Short-term (Current Implementation) - -1. **Monitor Performance:** - - Add logging for trigger execution time - - Monitor search_vector size distribution - - Track search query performance - -2. **Index Verification:** - - Ensure `custom_field_values_member_id_idx` exists and is used - - Verify GIN index on `search_vector` is maintained - -3. **Bulk Operations:** - - For bulk imports, consider temporarily disabling the custom_field_values trigger - - Re-enable and update search_vectors in batch after import - -### Medium-term Optimizations - -1. **✅ Optimize Trigger Function (FULLY IMPLEMENTED):** - - ✅ Only fetch required member fields instead of full record (reduces overhead) - - ✅ Skip re-aggregation on UPDATE if value hasn't actually changed (early return optimization) - -2. **Limit Search Vector Size:** - - Truncate very long custom field values (e.g., first 1000 chars) - - Add warning if aggregated text exceeds threshold - -3. **Optimize LIKE Queries:** - - Consider adding a generated column for searchable text - - Or use a materialized view for custom field search - -### Long-term Considerations - -1. **Alternative Approaches:** - - Separate search index table for custom fields - - Use Elasticsearch or similar for advanced search - - Materialized view for search optimization - -2. **Scaling Strategy:** - - If performance becomes an issue with 100+ custom fields per member: - - Consider limiting which custom fields are searchable - - Use a separate search service - - Implement search result caching - -## Performance Benchmarks (Estimated) - -Based on typical PostgreSQL performance: - -| Scenario | Members | Custom Fields/Member | Expected Impact | -|----------|---------|---------------------|-----------------| -| Small | < 1,000 | < 10 | Negligible (< 5ms per operation) | -| Medium | 1,000-10,000 | 10-30 | Minor (5-20ms per operation) | -| Large | 10,000-100,000 | 30-50 | Noticeable (20-50ms per operation) | -| Very Large | > 100,000 | 50+ | Significant (50-200ms+ per operation) | +Priority: **FTS > Substring > Custom Fields > Fuzzy**. ## Monitoring Queries ```sql --- Check search_vector size distribution -SELECT - pg_size_pretty(octet_length(search_vector::text)) as size, - COUNT(*) as member_count +-- search_vector size distribution +SELECT + pg_size_pretty(octet_length(search_vector::text)) AS size, + COUNT(*) AS member_count FROM members WHERE search_vector IS NOT NULL GROUP BY octet_length(search_vector::text) ORDER BY octet_length(search_vector::text) DESC LIMIT 20; --- Check average custom fields per member -SELECT - AVG(custom_field_count) as avg_custom_fields, - MAX(custom_field_count) as max_custom_fields +-- average / max custom fields per member +SELECT + AVG(custom_field_count) AS avg_custom_fields, + MAX(custom_field_count) AS max_custom_fields FROM ( - SELECT member_id, COUNT(*) as custom_field_count + SELECT member_id, COUNT(*) AS custom_field_count FROM custom_field_values GROUP BY member_id ) subq; --- Check trigger execution time (requires pg_stat_statements) -SELECT - mean_exec_time, - calls, - query +-- trigger execution time (requires pg_stat_statements) +SELECT mean_exec_time, calls, query FROM pg_stat_statements WHERE query LIKE '%members_search_vector_trigger%' ORDER BY mean_exec_time DESC; ``` -## Code Quality Improvements (Post-Review) - -### Refactored Search Implementation - -The search query has been refactored for better maintainability and clarity: - -**Before:** Single large OR-chain with mixed search types (hard to maintain) - -**After:** Modular functions grouped by search type: -- `build_fts_filter/1` - Full-text search (highest priority, fastest) -- `build_substring_filter/2` - Substring matching on structured fields -- `build_custom_field_filter/1` - Custom field value search (JSONB LIKE) -- `build_fuzzy_filter/2` - Trigram/fuzzy matching for names and streets - -**Benefits:** -- ✅ Clear separation of concerns -- ✅ Easier to maintain and test -- ✅ Better documentation of search priority -- ✅ Easier to optimize individual search types - -**Search Priority Order:** -1. **FTS (Full-Text Search)** - Fastest, uses GIN index on search_vector -2. **Substring** - For structured fields (postal_code, phone_number, etc.) -3. **Custom Fields** - JSONB LIKE queries (fallback for substring matching) -4. **Fuzzy Matching** - Trigram similarity for names and streets - -## Conclusion - -The current implementation is **well-optimized for typical use cases** (< 30 custom fields per member, < 10,000 members). For larger scales, monitoring and potential optimizations may be needed. - -**Key Strengths:** -- Indexed lookups (member_id index) -- Efficient GIN index for search -- Trigger-based automatic updates -- Modular, maintainable search code structure - -**Key Weaknesses:** -- LIKE queries on JSONB (not indexed) -- Re-aggregation on every custom field change (necessary for consistency) -- Potential size issues with many/large custom fields -- Substring searches (contains/ILIKE) not index-optimized - -**Recent Optimizations:** -- ✅ Trigger function optimized to fetch only required fields (reduces overhead by ~30-50%) -- ✅ Early return on UPDATE when value hasn't changed (skips expensive re-aggregation, <1ms vs 3-10ms) -- ✅ Improved performance for custom field value updates (3-10ms vs 5-15ms when value changes) +## Future Options (if scale demands) +- Generated/searchable text column or materialized view for custom-field substring search (to escape the unindexed JSONB `LIKE`). +- Limit which custom fields are searchable, or truncate long values. +- External search service (e.g., Elasticsearch) for advanced search. diff --git a/docs/database-schema-readme.md b/docs/database-schema-readme.md index fa6ea55..a7bfb1a 100644 --- a/docs/database-schema-readme.md +++ b/docs/database-schema-readme.md @@ -4,105 +4,54 @@ This document provides a comprehensive overview of the Mila Membership Management System database schema. -## Quick Links +- **DBML file:** [`database_schema.dbml`](./database_schema.dbml) — full per-column intent notes and relationship edges. +- **Search-vector performance:** see [`custom-fields-search-performance.md`](./custom-fields-search-performance.md) for trigger cost analysis and tuning. -- **DBML File:** [`database_schema.dbml`](./database_schema.dbml) -- **Visualize Online:** - - [dbdiagram.io](https://dbdiagram.io) - Upload the DBML file - - [dbdocs.io](https://dbdocs.io) - Generate interactive documentation +The DBML is **hand-maintained** (not auto-generated); keep it in sync with `priv/repo/migrations/`. ## Schema Statistics | Metric | Count | |--------|-------| -| **Tables** | 11 | +| **Tables** | 12 | | **Domains** | 4 (Accounts, Membership, MembershipFees, Authorization) | -| **Relationships** | 9 | -| **Indexes** | 25+ | -| **Triggers** | 1 (Full-text search) | +| **Triggers** | 3 (member, custom_field_values, member_groups → member search-vector) | ## Tables Overview ### Accounts Domain +- **`users`** — authentication accounts. Dual auth (Password + OIDC), optional 1:1 link to a member; email is the source of truth when linked. +- **`tokens`** — JWT storage for AshAuthentication; multiple purposes, revocation by deletion. -#### `users` -- **Purpose:** User authentication and session management -- **Rows (Estimated):** Low to Medium (typically 10-50% of members) -- **Key Features:** - - Dual authentication (Password + OIDC) - - Optional 1:1 link to members - - Email as source of truth when linked - -#### `tokens` -- **Purpose:** JWT token storage for AshAuthentication -- **Rows (Estimated):** Medium to High (multiple tokens per user) -- **Key Features:** - - Token lifecycle management - - Revocation support - - Multiple token purposes +OIDC account linking is recorded on the `users` table via the `oidc_id` column; there is no separate `user_identities` table. ### Membership Domain - -#### `members` -- **Purpose:** Club member master data -- **Rows (Estimated):** High (core entity) -- **Key Features:** - - Complete member profile - - Full-text search via tsvector - - Bidirectional email sync with users - - Flexible address and contact data - -#### `custom_field_values` -- **Purpose:** Dynamic custom member attributes -- **Rows (Estimated):** Variable (N per member) -- **Key Features:** - - Union type value storage (JSONB) - - Multiple data types supported - - One custom field value per custom field per member - -#### `custom_fields` -- **Purpose:** Schema definitions for custom_field_values -- **Rows (Estimated):** Low (admin-defined) -- **Key Features:** - - Type definitions - - Immutable and required flags - - Centralized custom field management - -#### `settings` -- **Purpose:** Global application settings (singleton resource) -- **Rows (Estimated):** 1 (singleton pattern) -- **Key Features:** - - Club name configuration - - Member field visibility settings - - Membership fee default settings - - Environment variable support for club name - -#### `groups` -- **Purpose:** Group definitions for organizing members -- **Rows (Estimated):** Low (typically 5-20 groups per club) -- **Key Features:** - - Unique group names (case-insensitive) - - URL-friendly slugs (auto-generated, immutable) - - Optional descriptions - - Many-to-many relationship with members - -#### `member_groups` -- **Purpose:** Join table for many-to-many relationship between members and groups -- **Rows (Estimated):** Medium to High (multiple groups per member) -- **Key Features:** - - Unique constraint on (member_id, group_id) - - CASCADE delete on both sides - - Efficient indexes for queries +- **`members`** — club member master data. Full-text + fuzzy search, bidirectional email sync with users, flexible address/contact data, `country`, optional `vereinfacht_contact_id` (external vereinfacht.de contact). +- **`custom_field_values`** — dynamic per-member attributes. Union-type value in JSONB; one value per custom field per member. +- **`custom_fields`** — schema definitions for custom field values (type, `required`/`show_in_overview` flags, optional `join_description`, auto-generated slug). +- **`settings`** — global application settings (singleton). Club name (also via `ASSOCIATION_NAME` env), member-field visibility/required maps, fee defaults, plus OIDC, SMTP/mail-from, vereinfacht.de, public join-form, `registration_enabled`, and `oidc_only` configuration. See [Settings configuration columns](#settings-configuration-columns). +- **`groups`** — member groupings. Case-insensitive-unique names, auto-generated immutable slugs, optional descriptions; many-to-many with members. +- **`member_groups`** — join table for members ↔ groups. Unique `(member_id, group_id)`, CASCADE delete on both sides (join table only). +- **`join_requests`** — public join flow (onboarding, double opt-in). Status machine `pending_confirmation → submitted → approved/rejected`; confirmation token stored as hash only, ~24h retention for unconfirmed records. ### Authorization Domain +- **`roles`** — RBAC. Links users to one of four hardcoded permission sets (`own_data`, `read_only`, `normal_user`, `admin`); system roles are deletion-protected. -#### `roles` -- **Purpose:** Role-based access control (RBAC) -- **Rows (Estimated):** Low (typically 3-10 roles) -- **Key Features:** - - Links users to permission sets - - System role protection - - Four hardcoded permission sets: own_data, read_only, normal_user, admin +### MembershipFees Domain +- **`membership_fee_types`** — fee types with immutable billing interval. +- **`membership_fee_cycles`** — per-member billing cycles with payment status. + +## Settings configuration columns + +The singleton `settings` row carries runtime configuration (all nullable unless noted). Grouped by area: + +- **Member overview:** `member_field_visibility` (JSONB; absent key = visible), `member_field_required` (JSONB). +- **Membership fees:** `include_joining_cycle` (bool, NOT NULL, default true), `default_membership_fee_type_id` (FK → membership_fee_types, ON DELETE SET NULL). +- **Registration / login:** `registration_enabled` (bool, NOT NULL, default true), `oidc_only` (bool, NOT NULL, default false). +- **OIDC:** `oidc_client_id`, `oidc_client_secret`, `oidc_base_url`, `oidc_redirect_uri`, `oidc_admin_group_name`, `oidc_groups_claim`. +- **SMTP / mail-from:** `smtp_host`, `smtp_port` (bigint), `smtp_username`, `smtp_password`, `smtp_ssl`, `smtp_from_name`, `smtp_from_email`. +- **vereinfacht.de:** `vereinfacht_api_url`, `vereinfacht_api_key`, `vereinfacht_club_id`, `vereinfacht_app_url`. +- **Public join form:** `join_form_enabled` (bool, NOT NULL, default false), `join_form_field_ids` (text[]), `join_form_field_required` (JSONB). ## Key Relationships @@ -124,123 +73,54 @@ Member (N) ←→ (N) Group Settings (1) → MembershipFeeType (0..1) ``` -### Relationship Details +## Foreign Key On-Delete Behavior -1. **User ↔ Member (Optional 1:1, both sides optional)** - - A User can have 0 or 1 Member (`user.member_id` can be NULL) - - A Member can have 0 or 1 User (optional `has_one` relationship) - - Both entities can exist independently - - Email synchronization when linked (User.email is source of truth) - - `ON DELETE SET NULL` on user side (User preserved when Member deleted) +| Relationship | On Delete | Rationale | +|--------------|-----------|-----------| +| `users.member_id → members.id` | SET NULL | Preserve user account when member deleted | +| `users.role_id → roles.id` | RESTRICT | Cannot delete a role that still has users | +| `custom_field_values.member_id → members.id` | CASCADE | Delete values with member | +| `custom_field_values.custom_field_id → custom_fields.id` | CASCADE | Delete values when the custom field is deleted | +| `members.membership_fee_type_id → membership_fee_types.id` | RESTRICT | Cannot delete a fee type assigned to members | +| `membership_fee_cycles.member_id → members.id` | CASCADE | Cycles deleted with member | +| `membership_fee_cycles.membership_fee_type_id → membership_fee_types.id` | RESTRICT | Cannot delete a fee type with cycles | +| `settings.default_membership_fee_type_id → membership_fee_types.id` | SET NULL | Clear default if fee type deleted | +| `member_groups.member_id → members.id` | CASCADE | Association removed; member preserved | +| `member_groups.group_id → groups.id` | CASCADE | Association removed; group preserved | -2. **User → Role (N:1)** - - Many users can be assigned to one role - - `ON DELETE RESTRICT` - cannot delete role if users are assigned - - Role links user to permission set for authorization +`join_requests.reviewed_by_user_id` is intentionally **unconstrained** (no FK); `reviewed_by_display` is denormalized so the UI need not load the reviewer User. -3. **Member → CustomFieldValues (1:N)** - - One member, many custom_field_values - - `ON DELETE CASCADE` - custom_field_values deleted with member - - Composite unique constraint (member_id, custom_field_id) - -4. **CustomFieldValue → CustomField (N:1)** - - Custom field values reference type definition - - `ON DELETE RESTRICT` - cannot delete type if in use - - Type defines data structure - -5. **Member → MembershipFeeType (N:1, optional)** - - Many members can be assigned to one fee type - - `ON DELETE RESTRICT` - cannot delete fee type if members are assigned - - Optional relationship (member can have no fee type) - -6. **Member → MembershipFeeCycles (1:N)** - - One member, many billing cycles - - `ON DELETE CASCADE` - cycles deleted when member deleted - - Unique constraint (member_id, cycle_start) - -7. **MembershipFeeCycle → MembershipFeeType (N:1)** - - Many cycles reference one fee type - - `ON DELETE RESTRICT` - cannot delete fee type if cycles exist - -8. **Settings → MembershipFeeType (N:1, optional)** - - Settings can reference a default fee type - - `ON DELETE SET NULL` - if fee type is deleted, setting is cleared - -9. **Member ↔ Group (N:N via MemberGroup)** - - Many-to-many relationship through `member_groups` join table - - `ON DELETE CASCADE` on both sides - removing member/group removes associations - - Unique constraint on (member_id, group_id) prevents duplicates - - Groups searchable via member search vector +**User ↔ Member** is an optional 1:1 (both sides may be NULL; entities exist independently). **Member ↔ Group** is many-to-many through `member_groups` (CASCADE lives only on the join table). ## Important Business Rules ### Email Synchronization -- **User.email** is the source of truth when linked -- On linking: Member.email ← User.email (overwrite) -- After linking: Changes sync bidirectionally -- Validation prevents email conflicts +- **User.email is the source of truth when linked.** On linking, `Member.email ← User.email` (overwrite). Afterwards changes sync bidirectionally. Validation prevents email conflicts with other unlinked users. ### Authentication Strategies -- **Password:** Email + hashed_password -- **OIDC:** Email + oidc_id (Rauthy provider) -- At least one method required per user +- **Password:** email + hashed_password. **OIDC:** email + oidc_id (Rauthy provider), the external identity recorded via the `oidc_id` column on `users`. At least one method required per user. ### Member Constraints -- First name and last name required (min 1 char) -- Email unique, validated format (5-254 chars) -- Exit date must be after join date -- Phone: `+?[0-9\- ]{6,20}` -- Postal code: optional (no format validation) -- Country: optional +- `first_name` / `last_name`: optional, but if present min 1 char. +- `email`: unique, validated format (5–254 chars). +- `exit_date` must be after `join_date`. +- `postal_code`, `country`: optional, no format validation. ### CustomFieldValue System -- Maximum one custom field value per custom field per member -- Value stored as union type in JSONB -- Supported types: string, integer, boolean, date, email -- Types can be marked as immutable or required - -## Indexes - -### Performance Indexes - -**members:** -- `search_vector` (GIN) - Full-text search (tsvector) -- `first_name` (GIN trgm) - Fuzzy search on first name -- `last_name` (GIN trgm) - Fuzzy search on last name -- `email` (GIN trgm) - Fuzzy search on email -- `city` (GIN trgm) - Fuzzy search on city -- `street` (GIN trgm) - Fuzzy search on street -- `notes` (GIN trgm) - Fuzzy search on notes -- `email` (B-tree) - Exact email lookups -- `last_name` (B-tree) - Name sorting -- `join_date` (B-tree) - Date filtering - -**custom_field_values:** -- `member_id` - Member custom field value lookups -- `custom_field_id` - Type-based queries -- Composite `(member_id, custom_field_id)` - Uniqueness - -**tokens:** -- `subject` - User token lookups -- `expires_at` - Token cleanup -- `purpose` - Purpose-based queries - -**users:** -- `email` (unique) - Login lookups -- `oidc_id` (unique) - OIDC authentication -- `member_id` (unique) - Member linkage +- One value per custom field per member. Value stored as a union type in JSONB: `{type: "string|integer|boolean|date|email", value: }`. Custom fields can be marked `required` and toggled `show_in_overview`. ## Full-Text Search ### Implementation -- **Trigger** on `members` (INSERT/UPDATE): runs function `members_search_vector_trigger()` +- **Trigger** on `members` (INSERT/UPDATE): `update_search_vector` runs function `members_search_vector_trigger()` +- **Trigger** on `custom_field_values` (INSERT/UPDATE/DELETE): `update_member_search_vector_on_custom_field_value_change` runs function `update_member_search_vector_from_custom_field_value()` - **Trigger** on `member_groups` (INSERT/UPDATE/DELETE): `update_member_search_vector_on_member_groups_change` runs function `update_member_search_vector_from_member_groups()` - **Index Type:** GIN (Generalized Inverted Index) ### Weighted Fields - **Weight A (highest):** first_name, last_name - **Weight B:** email, notes, group names (from member_groups → groups) -- **Weight C:** city, street, house_number, postal_code, country, custom_field_values +- **Weight C:** city, street, house_number, postal_code, custom_field_values - **Weight D (lowest):** join_date, exit_date ### Group Names in Search @@ -258,291 +138,46 @@ Custom field values are automatically included in the search vector: ### Usage Example ```sql -SELECT * FROM members +SELECT * FROM members WHERE search_vector @@ to_tsquery('simple', 'john & doe'); ``` ## Fuzzy Search (Trigram-based) -### Implementation -- **Extension:** `pg_trgm` (PostgreSQL Trigram) -- **Index Type:** GIN with `gin_trgm_ops` operator class -- **Similarity Threshold:** 0.2 (default, configurable) -- **Added:** November 2025 (PR #187, closes #162) +- **Extension:** `pg_trgm`; GIN indexes with `gin_trgm_ops` on `first_name`, `last_name`, `email`, `city`, `street`, `notes`. +- **Similarity threshold:** 0.2 (default, configurable) — balances precision/recall. +- **Added:** November 2025 (PR #187, closes #162). -### How It Works -Fuzzy search combines multiple search strategies: -1. **Full-text search** - Primary filter using tsvector -2. **Trigram similarity** - `similarity(field, query) > threshold` -3. **Word similarity** - `word_similarity(query, field) > threshold` -4. **Substring matching** - `LIKE` and `ILIKE` for exact substrings -5. **Modulo operator** - `query % field` for quick similarity check +Fuzzy search combines several strategies (applied as an OR-chain alongside full-text and substring matching): -### Indexed Fields for Fuzzy Search -- `first_name` - GIN trigram index -- `last_name` - GIN trigram index -- `email` - GIN trigram index -- `city` - GIN trigram index -- `street` - GIN trigram index -- `notes` - GIN trigram index +1. Full-text search — primary filter via tsvector. +2. Trigram similarity — `similarity(field, query) > threshold`. +3. Word similarity — `word_similarity(query, field) > threshold`. +4. Substring matching — `LIKE` / `ILIKE`. +5. `%` operator — quick trigram-similarity check. -### Usage Example (Ash Action) -```elixir -# In LiveView or context -Member.fuzzy_search(Member, query: "john", similarity_threshold: 0.2) - -# Or using Ash Query directly -Member -|> Ash.Query.for_read(:search, %{query: "john", similarity_threshold: 0.2}) -|> Mv.Membership.read!() -``` - -### Usage Example (SQL) -```sql --- Trigram similarity search -SELECT * FROM members -WHERE similarity(first_name, 'john') > 0.2 - OR similarity(last_name, 'doe') > 0.2 -ORDER BY similarity(first_name, 'john') DESC; - --- Word similarity (better for partial matches) -SELECT * FROM members -WHERE word_similarity('john', first_name) > 0.2; - --- Quick similarity check with % operator -SELECT * FROM members -WHERE 'john' % first_name; -``` - -### Performance Considerations -- **GIN indexes** speed up trigram operations significantly -- **Similarity threshold** of 0.2 balances precision and recall -- **Combined approach** (FTS + trigram) provides best results -- Lower threshold = more results but less specific +For the Elixir search action and per-strategy filter functions, see `lib/membership/member.ex` and [`custom-fields-search-performance.md`](./custom-fields-search-performance.md). ## Database Extensions -### Required PostgreSQL Extensions +Installed extensions are defined in `Mv.Repo.installed_extensions/0`: -1. **uuid-ossp** - - Purpose: UUID generation functions - - Used for: `gen_random_uuid()`, `uuid_generate_v7()` +| Extension | Purpose | Notes | +|-----------|---------|-------| +| `ash-functions` | Ash helper SQL functions | installed by Ash | +| `citext` | Case-insensitive text | `users.email` | +| `pg_trgm` | Trigram fuzzy search | added in `20251001141005_add_trigram_to_members.exs`; operators `%`, `similarity()`, `word_similarity()` | -2. **citext** - - Purpose: Case-insensitive text type - - Used for: `users.email` (case-insensitive email matching) +`gen_random_uuid()` is built into PostgreSQL; `uuid_generate_v7()` is a custom SQL function defined in a migration (not provided by an extension). -3. **pg_trgm** - - Purpose: Trigram-based fuzzy text search and similarity matching - - Used for: Fuzzy member search with similarity scoring - - Operators: `%` (similarity), `word_similarity()`, `similarity()` - - Added in: Migration `20251001141005_add_trigram_to_members.exs` +## Sensitive Data (GDPR / logging) -### Installation -```sql -CREATE EXTENSION IF NOT EXISTS "uuid-ossp"; -CREATE EXTENSION IF NOT EXISTS "citext"; -CREATE EXTENSION IF NOT EXISTS "pg_trgm"; -``` - -## Migration Strategy - -### Ash Migrations -This project uses Ash Framework's migration system: - -```bash -# Generate new migration -mix ash.codegen --name add_new_feature - -# Apply migrations -mix ash.setup - -# Rollback migrations -mix ash_postgres.rollback -n 1 -``` - -### Migration Files Location -``` -priv/repo/migrations/ -├── 20250421101957_initialize_extensions_1.exs -├── 20250528163901_initial_migration.exs -├── 20250617090641_member_fields.exs -├── 20250620110850_add_accounts_domain.exs -├── 20250912085235_AddSearchVectorToMembers.exs -├── 20250926180341_add_unique_email_to_members.exs -├── 20251001141005_add_trigram_to_members.exs -└── 20251016130855_add_constraints_for_user_member_and_property.exs -``` - -## Data Integrity - -### Foreign Key Behaviors - -| Relationship | On Delete | Rationale | -|--------------|-----------|-----------| -| `users.member_id → members.id` | SET NULL | Preserve user account when member deleted | -| `custom_field_values.member_id → members.id` | CASCADE | Delete custom_field_values with member | -| `custom_field_values.custom_field_id → custom_fields.id` | RESTRICT | Prevent deletion of types in use | - -### Validation Layers - -1. **Database Level:** - - CHECK constraints - - NOT NULL constraints - - UNIQUE indexes - - Foreign key constraints - -2. **Application Level (Ash):** - - Custom validators - - Email format validation (EctoCommons.EmailValidator) - - Business rule validation - - Cross-entity validation - -3. **UI Level:** - - Client-side form validation - - Real-time feedback - - Error messages - -## Performance Considerations - -### Query Patterns - -**High Frequency:** -- Member search (uses GIN index on search_vector) -- Member list with filters (uses indexes on join_date, membership_fee_type_id) -- User authentication (uses unique index on email/oidc_id) -- CustomFieldValue lookups by member (uses index on member_id) - -**Medium Frequency:** -- Member CRUD operations -- CustomFieldValue updates -- Token validation - -**Low Frequency:** -- CustomField management -- User-Member linking -- Bulk operations - -### Optimization Tips - -1. **Use indexes:** All critical query paths have indexes -2. **Preload relationships:** Use Ash's `load` to avoid N+1 -3. **Pagination:** Use keyset pagination (configured by default) -4. **GIN indexes:** Full-text search and fuzzy search on multiple fields -5. **Search optimization:** Full-text search via tsvector, not LIKE - -## Visualization - -### Using dbdiagram.io - -1. Visit [https://dbdiagram.io](https://dbdiagram.io) -2. Click "Import" → "From file" -3. Upload `database_schema.dbml` -4. View interactive diagram with relationships - -### Using dbdocs.io - -1. Install dbdocs CLI: `npm install -g dbdocs` -2. Generate docs: `dbdocs build database_schema.dbml` -3. View generated documentation - -### VS Code Extension - -Install "DBML Language" extension to view/edit DBML files with: -- Syntax highlighting -- Inline documentation -- Error checking - -## Security Considerations - -### Sensitive Data - -**Encrypted:** -- `users.hashed_password` (bcrypt) - -**Should Not Log:** -- hashed_password -- tokens (jti, purpose, extra_data) - -**Personal Data (GDPR):** -- All member fields (name, email, address) -- User email -- Token subject - -### Access Control - -- Implement through Ash policies -- Row-level security considerations for future -- Audit logging for sensitive operations - -## Backup Recommendations - -### Critical Tables (Priority 1) -- `members` - Core business data -- `users` - Authentication data -- `custom_fields` - Schema definitions - -### Important Tables (Priority 2) -- `custom_field_values` - Member custom data -- `tokens` - Can be regenerated but good to backup - -### Backup Strategy -```bash -# Full database backup -pg_dump -Fc mv_prod > backup_$(date +%Y%m%d).dump - -# Restore -pg_restore -d mv_prod backup_20251110.dump -``` - -## Testing - -### Test Database -- Separate test database: `mv_test` -- Sandbox mode via Ecto.Adapters.SQL.Sandbox -- Reset between tests - -### Seed Data -```bash -# Load seed data -mix run priv/repo/seeds.exs -``` - -## Future Considerations - -### Potential Additions - -1. **Audit Log Table** - - Track changes to members - - Compliance and history tracking - -2. **Payment Tracking** - - Payment history table - - Transaction records - - Fee calculation - -3. **Document Storage** - - Member documents/attachments - - File metadata table - -4. **Email Queue** - - Outbound email tracking - - Delivery status - -5. **Roles & Permissions** - - User roles (admin, treasurer, member) - - Permission management - -## Resources - -- **Ash Framework:** [https://hexdocs.pm/ash](https://hexdocs.pm/ash) -- **AshPostgres:** [https://hexdocs.pm/ash_postgres](https://hexdocs.pm/ash_postgres) -- **DBML Specification:** [https://dbml.dbdiagram.io](https://dbml.dbdiagram.io) -- **PostgreSQL Docs:** [https://www.postgresql.org/docs/](https://www.postgresql.org/docs/) +- **Never log:** `users.hashed_password` (bcrypt), token fields (`jti`, `purpose`, `extra_data`), OIDC/SMTP/vereinfacht secrets in `settings`. +- **Personal data:** all member fields, user email, join-request applicant data. --- -**Last Updated:** 2026-01-27 -**Schema Version:** 1.5 +**Last Updated:** 2026-06-15 +**Schema Version:** 1.6 (12 tables) **Database:** PostgreSQL 17.6 (dev) / 16 (prod) diff --git a/docs/database_schema.dbml b/docs/database_schema.dbml index 16c9723..f763726 100644 --- a/docs/database_schema.dbml +++ b/docs/database_schema.dbml @@ -6,8 +6,9 @@ // - https://dbdocs.io // - VS Code Extensions: "DBML Language" or "dbdiagram.io" // -// Version: 1.4 -// Last Updated: 2026-01-13 +// Version: 1.6 +// Last Updated: 2026-06-15 +// Hand-maintained (NOT auto-generated). 12 tables. Project mila_membership_management { database_type: 'PostgreSQL' @@ -25,15 +26,16 @@ Project mila_membership_management { - GDPR-compliant data management ## Domains: - - **Accounts**: User authentication and session management - - **Membership**: Club member data and custom fields + - **Accounts**: User authentication, sessions, OIDC strategy identities + - **Membership**: Club member data, custom fields, groups, settings, public join requests - **MembershipFees**: Membership fee types and billing cycles - **Authorization**: Role-based access control (RBAC) - ## Required PostgreSQL Extensions: - - uuid-ossp (UUID generation) + ## Required PostgreSQL Extensions (see Mv.Repo.installed_extensions/0): + - ash-functions (Ash helper SQL functions) - citext (case-insensitive text) - pg_trgm (trigram-based fuzzy search) + UUIDv7 ids use uuid_generate_v7(), a custom SQL function defined in a migration (not an extension). ''' } @@ -135,7 +137,8 @@ Table members { search_vector tsvector [null, note: 'Full-text search index (auto-generated)'] membership_fee_type_id uuid [null, note: 'FK to membership_fee_types - assigned fee type'] membership_fee_start_date date [null, note: 'Date from which membership fees should be calculated'] - + vereinfacht_contact_id text [null, note: 'External contact id from the vereinfacht.de API (no FK; null if unlinked)'] + indexes { email [unique, name: 'members_unique_email_index'] search_vector [type: gin, name: 'members_search_vector_idx', note: 'GIN index for full-text search (tsvector)'] @@ -169,7 +172,8 @@ Table members { **Search Capabilities:** 1. Full-Text Search (tsvector): - `search_vector` is auto-updated via trigger - - Weighted fields: first_name (A), last_name (A), email (B), notes (B) + - Weighted fields (A/B/C/D map): see the "Weighted Fields" section of + database-schema-readme.md (single source of truth, matches the search trigger) - GIN index for fast text search 2. Fuzzy Search (pg_trgm): @@ -225,7 +229,7 @@ Table custom_field_values { **Constraints:** - Each member can have only ONE custom field value per custom field - Custom field values are deleted when member is deleted (CASCADE) - - Custom field cannot be deleted if custom field values exist (RESTRICT) + - Custom field values are deleted when the custom field is deleted (CASCADE) **Use Cases:** - Custom membership numbers @@ -241,8 +245,9 @@ Table custom_fields { slug text [not null, unique, note: 'URL-friendly, immutable identifier (e.g., "membership-number"). Auto-generated from name.'] value_type text [not null, note: 'Data type: string | integer | boolean | date | email'] description text [null, note: 'Human-readable description'] - immutable boolean [not null, default: false, note: 'If true, value cannot be changed after creation'] + join_description text [null, note: 'Optional label shown for this field on the public join form (e.g., a GDPR confirmation text); supports inline external links. Falls back to name when null.'] required boolean [not null, default: false, note: 'If true, all members must have this custom field'] + show_in_overview boolean [not null, default: true, note: 'If true, this custom field is displayed in the member overview table and can be sorted'] indexes { name [unique, name: 'custom_fields_unique_name_index'] @@ -259,8 +264,9 @@ Table custom_fields { - `slug`: URL-friendly, human-readable identifier (auto-generated, immutable) - `value_type`: Enforces data type consistency - `description`: Documentation for users/admins - - `immutable`: Prevents changes after initial creation (e.g., membership numbers) + - `join_description`: Optional label shown for this field on the public join form (falls back to `name` when null) - `required`: Enforces that all members must have this custom field + - `show_in_overview`: When true, the field is shown in the member overview table and can be sorted **Slug Generation:** - Automatically generated from `name` on creation @@ -275,13 +281,13 @@ Table custom_fields { - `name` must be unique across all custom fields - `slug` must be unique across all custom fields - `slug` cannot be empty (validated on creation) - - Cannot be deleted if custom_field_values reference it (ON DELETE RESTRICT) - + - Deleting a custom field cascades: its custom_field_values are deleted too (ON DELETE CASCADE) + **Examples:** - - Membership Number (string, immutable, required) → slug: "membership-number" - - Emergency Contact (string, mutable, optional) → slug: "emergency-contact" - - Certified Trainer (boolean, mutable, optional) → slug: "certified-trainer" - - Certification Date (date, immutable, optional) → slug: "certification-date" + - Membership Number (string, required) → slug: "membership-number" + - Emergency Contact (string, optional) → slug: "emergency-contact" + - Certified Trainer (boolean, optional) → slug: "certified-trainer" + - Certification Date (date, optional) → slug: "certification-date" ''' } @@ -399,8 +405,8 @@ Ref: custom_field_values.member_id > members.id [delete: cascade] // CustomFieldValue → CustomField (N:1) // - Many custom_field_values can reference one custom field // - CustomFieldValue type defines the schema/behavior -// - ON DELETE RESTRICT: Cannot delete type if custom_field_values exist -Ref: custom_field_values.custom_field_id > custom_fields.id [delete: restrict] +// - ON DELETE CASCADE: deleting the custom field deletes its custom_field_values +Ref: custom_field_values.custom_field_id > custom_fields.id [delete: cascade] // Member → MembershipFeeType (N:1) // - Many members can be assigned to one fee type @@ -462,25 +468,14 @@ Enum membership_fee_status { TableGroup accounts_domain { users tokens - + Note: ''' **Accounts Domain** - - Handles user authentication and session management using AshAuthentication. - Supports multiple authentication strategies (Password, OIDC). - ''' -} -TableGroup membership_domain { - members - custom_field_values - custom_fields - - Note: ''' - **Membership Domain** - - Core business logic for club membership management. - Supports flexible, extensible member data model. + Handles user authentication and session management using AshAuthentication. + Supports multiple authentication strategies (Password, OIDC). OIDC linking + is recorded on the users table via the oidc_id column (there is no separate + user_identities table). ''' } @@ -550,9 +545,32 @@ Table roles { Table settings { id uuid [pk, not null, default: `gen_random_uuid()`, note: 'Primary identifier'] club_name text [not null, note: 'The name of the association/club (min length: 1)'] - member_field_visibility jsonb [null, note: 'Visibility configuration for member fields in overview (JSONB map)'] + member_field_visibility jsonb [null, note: 'Visibility config for member fields in overview (JSONB map; absent key = visible)'] + member_field_required jsonb [null, note: 'Required-field config for member fields (JSONB map)'] include_joining_cycle boolean [not null, default: true, note: 'Whether to include the joining cycle in membership fee generation'] - default_membership_fee_type_id uuid [null, note: 'FK to membership_fee_types - default fee type for new members'] + default_membership_fee_type_id uuid [null, note: 'Logical reference to membership_fee_types (default fee type for new members) - app-enforced, NO DB foreign key'] + registration_enabled boolean [not null, default: true, note: 'Whether self-service user registration is enabled'] + oidc_only boolean [not null, default: false, note: 'If true, only OIDC login is offered (password login hidden)'] + oidc_client_id text [null, note: 'OIDC client id'] + oidc_client_secret text [null, note: 'OIDC client secret'] + oidc_base_url text [null, note: 'OIDC provider base URL (e.g., Rauthy)'] + oidc_redirect_uri text [null, note: 'OIDC redirect URI'] + oidc_admin_group_name text [null, note: 'Provider group name mapped to admin role on login'] + oidc_groups_claim text [null, note: 'JWT claim carrying the user groups for role sync'] + smtp_host text [null, note: 'Outbound SMTP host'] + smtp_port bigint [null, note: 'Outbound SMTP port'] + smtp_username text [null, note: 'SMTP auth username'] + smtp_password text [null, note: 'SMTP auth password (secret)'] + smtp_ssl text [null, note: 'SMTP TLS/SSL mode'] + smtp_from_name text [null, note: 'Display name for the From header (mail_from)'] + smtp_from_email text [null, note: 'Email address for the From header (mail_from)'] + vereinfacht_api_url text [null, note: 'vereinfacht.de API base URL'] + vereinfacht_api_key text [null, note: 'vereinfacht.de API key (secret)'] + vereinfacht_club_id text [null, note: 'vereinfacht.de club identifier'] + vereinfacht_app_url text [null, note: 'vereinfacht.de app URL (for links)'] + join_form_enabled boolean [not null, default: false, note: 'Whether the public join form is enabled'] + join_form_field_ids text[] [null, note: 'Ordered custom_field ids shown on the public join form'] + join_form_field_required jsonb [null, note: 'Per-field required config for the join form (JSONB map)'] inserted_at timestamp [not null, default: `now() AT TIME ZONE 'utc'`, note: 'Creation timestamp (UTC)'] updated_at timestamp [not null, default: `now() AT TIME ZONE 'utc'`, note: 'Last update timestamp (UTC)'] @@ -590,19 +608,123 @@ Table settings { ''' } +// ============================================ +// MEMBERSHIP DOMAIN — Groups +// ============================================ + +Table groups { + id uuid [pk, not null, default: `uuid_generate_v7()`, note: 'UUIDv7 primary key'] + name text [not null, note: 'Group name (unique case-insensitively via LOWER(name))'] + slug text [not null, unique, note: 'URL-friendly, immutable identifier auto-generated from name (shared GenerateSlug change)'] + description text [null, note: 'Optional description'] + inserted_at timestamp [not null, default: `now() AT TIME ZONE 'utc'`, note: 'Creation timestamp (UTC)'] + updated_at timestamp [not null, default: `now() AT TIME ZONE 'utc'`, note: 'Last update timestamp (UTC)'] + + indexes { + slug [unique, name: 'groups_unique_slug_index', note: 'Case-sensitive unique slug'] + name [unique, name: 'groups_unique_name_lower_index', note: 'UNIQUE on LOWER(name) - case-insensitive name uniqueness'] + } + + Note: ''' + **Member Groups** + + Flat groupings of members (no hierarchy in current schema). Many-to-many + with members via member_groups. `slug` is generated by the shared + Mv.Membership.Changes.GenerateSlug change (same as custom_fields) and is + used for URL routing (/groups/:slug). Group names feed the member + search_vector at weight B (see member_groups note). + + **Future extension path (not yet in schema):** + - parent_group_id (self-referential, nullable) + circular-ref guard + path calc for hierarchy + - member_group_roles table linking MemberGroup to a Role (position within a group) + ''' +} + +Table member_groups { + id uuid [pk, not null, default: `uuid_generate_v7()`, note: 'UUIDv7 primary key'] + member_id uuid [not null, note: 'FK to members'] + group_id uuid [not null, note: 'FK to groups'] + inserted_at timestamp [not null, default: `now() AT TIME ZONE 'utc'`, note: 'Creation timestamp (UTC)'] + updated_at timestamp [not null, default: `now() AT TIME ZONE 'utc'`, note: 'Last update timestamp (UTC)'] + + indexes { + (member_id, group_id) [unique, name: 'member_groups_unique_member_group_index', note: 'One association per member per group'] + member_id [name: 'member_groups_member_id_index'] + group_id [name: 'member_groups_group_id_index'] + } + + Note: ''' + **Member ↔ Group Join Table** + + CASCADE delete on BOTH foreign keys (the cascade lives only on the join + table; members and groups themselves are never deleted by association + removal). INSERT/UPDATE/DELETE here fires the trigger that refreshes the + affected member's search_vector so group names (weight B) stay current. + ''' +} + +// ============================================ +// MEMBERSHIP DOMAIN — Public Join Requests +// ============================================ + +Table join_requests { + id uuid [pk, not null, default: `uuid_generate_v7()`, note: 'UUIDv7 primary key'] + status text [not null, default: 'pending_confirmation', note: 'pending_confirmation → submitted (after email confirm) → approved/rejected'] + email text [not null, note: 'Applicant email'] + first_name text [null, note: 'Applicant first name'] + last_name text [null, note: 'Applicant last name'] + form_data jsonb [null, note: 'Submitted join-form field values (custom fields)'] + schema_version integer [null, note: 'Version of the join-form schema used at submission time'] + confirmation_token_hash text [null, note: 'Hash of the double-opt-in token (raw token never stored)'] + confirmation_token_expires_at timestamp [null, note: 'Token expiry (UTC)'] + confirmation_sent_at timestamp [null, note: 'When the confirmation email was sent (UTC)'] + submitted_at timestamp [null, note: 'When email was confirmed and request submitted (UTC)'] + approved_at timestamp [null, note: 'When an admin approved (UTC)'] + rejected_at timestamp [null, note: 'When an admin rejected (UTC)'] + reviewed_by_user_id uuid [null, note: 'User who reviewed (no FK constraint)'] + reviewed_by_display text [null, note: 'Reviewer display string, denormalized so the UI need not load the User'] + source text [null, note: 'Origin of the request (e.g., public form)'] + inserted_at timestamp [not null, default: `now() AT TIME ZONE 'utc'`, note: 'Creation timestamp (UTC)'] + updated_at timestamp [not null, default: `now() AT TIME ZONE 'utc'`, note: 'Last update timestamp (UTC)'] + + indexes { + confirmation_token_hash [unique, name: 'join_requests_confirmation_token_hash_unique', note: 'Partial unique WHERE confirmation_token_hash IS NOT NULL'] + email [name: 'join_requests_email_index'] + status [name: 'join_requests_status_index'] + } + + Note: ''' + **Public Join Flow (Onboarding, Double Opt-In)** + + Stores public join-form submissions. Double opt-in: the confirmation token + is stored as a hash only; unconfirmed records have a ~24h retention and are + removed by a scheduled cleanup job. `reviewed_by_user_id` is intentionally + unconstrained (no FK); `reviewed_by_display` is denormalized so showing the + reviewer does not require loading the User. + ''' +} + // ============================================ // RELATIONSHIPS (Additional) // ============================================ +// MemberGroup → Member (N:1) +// - ON DELETE CASCADE (join table only): association removed, member preserved +Ref: member_groups.member_id > members.id [delete: cascade] + +// MemberGroup → Group (N:1) +// - ON DELETE CASCADE (join table only): association removed, group preserved +Ref: member_groups.group_id > groups.id [delete: cascade] + // User → Role (N:1) // - Many users can be assigned to one role // - ON DELETE RESTRICT: Cannot delete role if users are assigned Ref: users.role_id > roles.id [delete: restrict] -// Settings → MembershipFeeType (N:1, optional) -// - Settings can reference a default membership fee type -// - ON DELETE SET NULL: If fee type is deleted, setting is cleared -Ref: settings.default_membership_fee_type_id > membership_fee_types.id [delete: set null] +// Settings → MembershipFeeType (N:1, optional) — LOGICAL relationship only +// - No DB foreign key (cross-domain dependency is deliberately avoided); +// referential integrity is enforced in the app (Mv.Membership.Setting) +Ref: settings.default_membership_fee_type_id > membership_fee_types.id // ============================================ // TABLE GROUPS (Updated) @@ -624,12 +746,16 @@ TableGroup membership_domain { custom_field_values custom_fields settings - + groups + member_groups + join_requests + Note: ''' **Membership Domain** - + Core business logic for club membership management. Supports flexible, extensible member data model. - Includes global application settings (singleton). + Includes member groups (many-to-many), global application settings + (singleton), and the public join-request flow. ''' } diff --git a/docs/groups-architecture.md b/docs/groups-architecture.md index ca1f07b..0959488 100644 --- a/docs/groups-architecture.md +++ b/docs/groups-architecture.md @@ -1,1223 +1,101 @@ # Groups - Technical Architecture -**Project:** Mila - Membership Management System **Feature:** Groups Management -**Version:** 1.0 -**Last Updated:** 2025-01-XX -**Status:** ✅ Implemented (authorization: see [roles-and-permissions-architecture.md](./roles-and-permissions-architecture.md)) +**Status:** Implemented (authorization: see [roles-and-permissions-architecture.md](./roles-and-permissions-architecture.md)) + +This document records the durable design of the Groups feature: data model, key decisions, integration points, accessibility rules, and the planned extension paths. The original implementation plan (estimations, vertical slices, per-issue acceptance criteria, testing/migration strategy) has been removed now that the feature has shipped. + +**Related:** [database-schema-readme.md](./database-schema-readme.md), [roles-and-permissions-architecture.md](./roles-and-permissions-architecture.md). --- -## Purpose +## Core Design Decisions -This document defines the technical architecture for the Groups feature. It focuses on architectural decisions, patterns, module structure, and integration points **without** concrete implementation details. +1. **Many-to-many:** members can belong to multiple groups and vice versa, via the `member_groups` join table (a separate Ash resource). +2. **Flat structure:** no hierarchy in the current schema; the design leaves a clear path to add it later (see [Future Extensibility](#future-extensibility)). +3. **Minimal attributes:** `name`, `description`, `slug`. The `slug` is auto-generated from `name`, immutable, URL-friendly. +4. **Cascade on the join table only:** deleting a group (or member) removes the `member_groups` associations but never deletes members/groups themselves. Group deletion requires explicit confirmation (typing the group name). +5. **Search integration:** group names are included in the member `search_vector` (not a separate search index). -**Related Documents:** +## Domain & Resources -- [database-schema-readme.md](./database-schema-readme.md) - Database documentation -- [roles-and-permissions-architecture.md](./roles-and-permissions-architecture.md) - Authorization system +Groups live in the **`Mv.Membership`** domain alongside Members and CustomFields. ---- +- `Mv.Membership.Group` (`lib/membership/group.ex`) — attributes `name`, `slug`, `description`; `has_many :member_groups`, `many_to_many :members`; `member_count` aggregate (`count :member_count, :member_groups`); `unique_slug` identity for slug lookups. Slug is generated by the shared **`Mv.Membership.Changes.GenerateSlug`** change (the same change CustomFields uses), generated on create and immutable on update. +- `Mv.Membership.MemberGroup` (`lib/membership/member_group.ex`) — join table; `belongs_to :member`, `belongs_to :group`; unique on `(member_id, group_id)`. Has `create`/`destroy` actions only (no `update`); group membership is managed by creating and destroying these join rows. +- `Mv.Membership.Member` (extended) — `has_many :member_groups`, `many_to_many :groups`. Group membership is managed through the `MemberGroup` join resource, not via dedicated Member actions. -## Table of Contents +## Data Model -1. [Architecture Principles](#architecture-principles) -2. [Domain Structure](#domain-structure) -3. [Data Architecture](#data-architecture) -4. [Business Logic Architecture](#business-logic-architecture) -5. [UI/UX Architecture](#uiux-architecture) -6. [Integration Points](#integration-points) -7. [Authorization](#authorization) -8. [Performance Considerations](#performance-considerations) -9. [Future Extensibility](#future-extensibility) -10. [Implementation Phases](#implementation-phases) +### `groups` +- `id` (UUIDv7), `name` (required), `slug` (required, immutable, auto-generated), `description` (optional), timestamps. +- Uniqueness: `name` unique case-insensitively (`UNIQUE` on `LOWER(name)`, index `groups_unique_name_lower_index`); `slug` unique case-sensitively (`groups_unique_slug_index`). ---- +### `member_groups` (join table) +- `id` (UUIDv7), `member_id`, `group_id`, timestamps. +- Unique `(member_id, group_id)` prevents duplicates; indexes on `member_id` and `group_id`. +- **CASCADE delete on both foreign keys** — the cascade is intentionally on the join table only. -## Architecture Principles +For exact columns/indexes see `database_schema.dbml`. -### Core Design Decisions +## Search Integration -1. **Many-to-Many Relationship:** - - Members can belong to multiple groups - - Groups can contain multiple members - - Implemented via join table (`member_groups`) as separate Ash resource +Group names are part of the member full-text search: -2. **Flat Structure (MVP):** - - Groups are initially flat (no hierarchy) - - Architecture designed to allow hierarchical extension later - - No parent/child relationships in MVP +- They are aggregated from `member_groups` joined to `groups` and added to `members.search_vector` at **weight B**. +- The trigger `update_member_search_vector_on_member_groups_change` runs `update_member_search_vector_from_member_groups()` on **INSERT/UPDATE/DELETE on `member_groups`** and refreshes the affected member's `search_vector`. +- Migration `20260217120000_add_group_names_to_member_search_vector.exs` (Issue #375). No Elixir search change is needed — searching a group name finds its members automatically. -3. **Minimal Attributes (MVP):** - - `name`, `description`, and `slug` in initial version - - `slug` is automatically generated from `name` (immutable, URL-friendly) - - Extensible for future attributes (dates, status, etc.) +## UI Surface (implemented) -4. **Cascade Deletion:** - - Deleting a group removes all member-group associations - - Members themselves are not deleted (CASCADE on join table only) - - Requires explicit confirmation with group name input +- **`/groups`** — index table (name, description, member count, actions), sorted by name at the DB level. Create button → `/groups/new`. +- **`/groups/:slug`** — detail: group info, member list, inline add-member combobox (search/autocomplete, excludes members already in the group), per-row remove (no confirmation), edit/delete. Add/remove are guarded by `:update` permission both in the UI and server-side in the event handlers. +- **`/groups/:slug/edit`** and **`/groups/new`** — separate form pages; slug not editable. Edit does auth in `mount/3` and loads the group once in `handle_params/3`. +- **Delete confirmation modal** — warns with member count (pluralized), requires typing the group name to enable delete (`phx-debounce="200"`), stays open on mismatch, authorizes server-side. +- **Member overview** — "Groups" column with badges; filter dropdown (persisted in URL query params); sort by group; group names searchable. +- **Member detail** — Groups shown as a data field in Personal Data (below Linked User), button-style links to `/groups/:slug`. -5. **Search Integration:** - - Groups searchable within member search (not separate search) - - Group names included in member search vector for full-text search +## Accessibility ---- - -## Domain Structure - -### Ash Domain: `Mv.Membership` - -**Purpose:** Groups are part of the Membership domain, alongside Members and CustomFields - -**New Resources:** - -- `Group` - Group definitions (name, description, slug) -- `MemberGroup` - Join table for many-to-many relationship between Members and Groups - -**Extended Resources:** - -- `Member` - Extended with `has_many :groups` relationship (through MemberGroup) - -### Module Organization - -``` -lib/ -├── membership/ -│ ├── membership.ex # Domain definition (extended) -│ ├── group.ex # Group resource -│ ├── member_group.ex # MemberGroup join table resource -│ └── member.ex # Extended with groups relationship -├── mv_web/ -│ └── live/ -│ ├── group_live/ -│ │ ├── index.ex # Groups management page -│ │ ├── form.ex # Create/edit group form -│ │ └── show.ex # Group detail view -│ └── member_live/ -│ ├── index.ex # Extended with group filtering/sorting -│ └── show.ex # Extended with group display -└── mv/ - └── membership/ - └── group/ # Future: Group-specific business logic - └── helpers.ex # Group-related helper functions -``` - ---- - -## Data Architecture - -### Database Schema - -#### `groups` Table - -**Attributes:** -- `id` - UUID v7 primary key -- `name` - Unique group name (required, max 100 chars) -- `slug` - URL-friendly identifier (required, max 100 chars, auto-generated from name) -- `description` - Optional description (max 500 chars) -- `inserted_at` / `updated_at` - Timestamps - -**Constraints:** -- `name` must be unique (case-insensitive, using LOWER(name)) -- `slug` must be unique (case-sensitive, exact match) -- `name` cannot be null -- `slug` cannot be null -- `name` max length: 100 characters -- `slug` max length: 100 characters -- `description` max length: 500 characters - -#### `member_groups` Table (Join Table) - -**Attributes:** -- `id` - UUID v7 primary key -- `member_id` - Foreign key to members (CASCADE delete) -- `group_id` - Foreign key to groups (CASCADE delete) -- `inserted_at` / `updated_at` - Timestamps - -**Constraints:** -- Unique constraint on `(member_id, group_id)` - prevents duplicate memberships -- CASCADE delete: Removing member removes all group associations -- CASCADE delete: Removing group removes all member associations - -**Indexes:** -- Index on `member_id` for efficient member → groups queries -- Index on `group_id` for efficient group → members queries - -### Ash Resources - -#### `Mv.Membership.Group` - -**Relationships:** -- `has_many :member_groups` - Relationship to MemberGroup join table -- `many_to_many :members` - Relationship to Members through MemberGroup - -**Calculations:** -- `member_count` - Integer calculation counting associated members - -**Actions:** -- `create` - Create new group (auto-generates slug from name) -- `read` - List/search groups (can query by slug via identity) -- `update` - Update group name/description (slug remains unchanged) -- `destroy` - Delete group (with confirmation) - -**Validations:** -- `name` required, unique (case-insensitive), max 100 chars -- `slug` required, unique (case-sensitive), max 100 chars, auto-generated, immutable -- `description` optional, max 500 chars - -**Identities:** -- `unique_slug` - Unique identity on `slug` for efficient lookups - -#### `Mv.Membership.MemberGroup` - -**Relationships:** -- `belongs_to :member` - Relationship to Member -- `belongs_to :group` - Relationship to Group - -**Actions:** -- `create` - Add member to group -- `read` - Query member-group associations -- `destroy` - Remove member from group - -**Validations:** -- Unique constraint on `(member_id, group_id)` - -#### `Mv.Membership.Member` (Extended) - -**New Relationships:** -- `has_many :member_groups` - Relationship to MemberGroup join table -- `many_to_many :groups` - Relationship to Groups through MemberGroup - -**New Actions:** -- `add_to_groups` - Add member to one or more groups -- `remove_from_groups` - Remove member from one or more groups - ---- - -## Business Logic Architecture - -### Group Management - -**Create Group:** -- Validate name uniqueness -- Automatically generate slug from name (using `GenerateSlug` change, same pattern as CustomFields) -- Validate slug uniqueness -- Return created group - -**Update Group:** -- Validate name uniqueness (if name changed) -- Update description -- Slug remains unchanged (immutable after creation) -- Return updated group - -**Delete Group:** -- Check if group has members (for warning display) -- Require explicit confirmation (group name input) -- Cascade delete all `member_groups` associations -- Group itself deleted - -### Member-Group Association - -**Add Member to Group:** -- Validate member exists -- Validate group exists -- Check for duplicate association -- Create `MemberGroup` record - -**Remove Member from Group:** -- Find `MemberGroup` record -- Delete association -- Member and group remain intact - -**Bulk Operations:** -- Add member to multiple groups in single transaction -- Remove member from multiple groups in single transaction - -### Search Integration - -**Member Search Enhancement:** -- Include group names in member search vector -- When searching for member, also search in associated group names -- Example: Searching for a group name finds all members in groups with that name - -**Implementation:** -- Extend `member.search_vector` trigger to include group names -- Update trigger on `member_groups` changes -- Use PostgreSQL `tsvector` for full-text search - ---- - -## UI/UX Architecture - -### Groups Management Page (`/groups`) - -**Route:** `/groups` - Groups management index page - -**Features:** -- List all groups in table (sorted by name via database query) -- Create new group button (navigates to `/groups/new`) -- Edit group via separate form page (`/groups/:slug/edit`) -- Delete group with confirmation modal -- Show member count per group - -**Table Columns:** -- Name (sortable, searchable) -- Description -- Member Count -- Actions (Edit, Delete) - -**Delete Confirmation Modal:** -- Warning: "X members are in this group" (with proper pluralization) -- Confirmation: "All member-group associations will be permanently deleted" -- Input field: Enter group name to confirm (with `phx-debounce="200"` for better UX) -- Delete button disabled until name matches -- Modal remains open on name mismatch (allows user to correct input) -- Cancel button -- Server-side authorization check in delete event handler (security best practice) - -### Member Overview Integration - -**New Column: "Groups"** -- Display group badges for each member -- Badge shows group name -- Multiple badges if member in multiple groups -- *(Optional)* Click badge to filter by that group (enhanced UX, can be added later) - -**Filtering:** -- Dropdown/select to filter by group -- "All groups" option (default) -- Filter persists in URL query params -- Works with existing search/sort - -**Sorting:** -- Sort by group name (members with groups first, then alphabetically) -- Sort by number of groups (members with most groups first) - -**Search:** -- Group names included in member search -- Searching group name shows all members in that group - -### Member Detail View Integration - -**New Section: "Groups"** -- List all groups member belongs to -- Display as badges or list -- Add/remove groups inline -- Link to group detail page - -### Group Detail View (`/groups/:slug`) - -**Route:** `/groups/:slug` - Group detail page (uses slug for URL-friendly routing) - -**Features:** -- Display group name and description -- List all members in group -- Link to member detail pages -- Add members to group (via inline combobox with search/autocomplete) -- Remove members from group (via remove button per member) -- Edit group button (navigates to `/groups/:slug/edit`) -- Delete group button (with confirmation modal) - -**Add Member Functionality:** -- "Add Member" button displayed above member table (only for users with `:update` permission) -- Opens inline add member area with member search/autocomplete (combobox) -- Search filters out members already in the group -- Selecting a member adds them to the group immediately -- Success/error flash messages provide feedback -- "Cancel" button closes the inline add member area without adding - -**Remove Member Functionality:** -- "Remove" button (icon button) for each member in table (only for users with `:update` permission) -- Clicking remove immediately removes member from group (no confirmation dialog) -- Success/error flash messages provide feedback - -**Note:** Uses slug for routing to provide URL-friendly, readable group URLs (e.g., `/groups/board-members`). - -### Group Form Pages - -**Create Form:** `/groups/new` -- Separate LiveView page for creating new groups -- Form with name and description fields -- Slug is auto-generated and not editable -- Redirects to `/groups` on success - -**Edit Form:** `/groups/:slug/edit` -- Separate LiveView page for editing existing groups -- Form pre-populated with current group data -- Slug is immutable (not displayed in form) -- Redirects to `/groups/:slug` on success -- `mount/3` performs authorization check, `handle_params/3` loads group once - -### Accessibility (A11y) Considerations - -**Requirements:** -- All UI elements must be keyboard accessible -- Screen readers must be able to navigate and understand the interface -- ARIA labels and roles must be properly set - -**Group Badges and Links in Member Overview / Detail:** -- Use `aria-label` to indicate group membership (e.g. "Member of group X"). Do not use `role="status"` on badges or links—that role is for live regions (screen reader announcements), not for navigation or static labels. -- Badge/link text or title should indicate group membership for screen readers. - -**Clickable Group Badge (for filtering) - Optional:** - -**Note:** This is an optional enhancement. The dropdown filter provides the same functionality. The clickable badge improves UX by showing the active filter visually and allowing quick removal. - -**Estimated effort:** 1.5-2.5 hours - -- Clickable badges must be proper button elements with `type="button"` -- Must include `aria-label` describing the filter action -- Icon for removal should have `aria-hidden="true"` - -**Group Filter Dropdown:** -- Select element must have appropriate `id`, `name`, and `aria-label` attributes -- Options should clearly indicate selected state - -**Screen Reader Announcements:** -- Use `role="status"` with `aria-live="polite"` for dynamic content -- Announce filter changes and member count updates - -**Delete Confirmation Modal:** -- Modal must use proper `role="dialog"` with `aria-labelledby` and `aria-describedby` -- Warning messages must be clearly associated with the modal description -- Form inputs must be properly labeled - -**Keyboard Navigation:** -- All interactive elements (buttons, links, form inputs) must be focusable via Tab key -- Modal dialogs must trap focus (Tab key cycles within modal) -- Escape key closes modals -- Enter/Space activates buttons when focused - ---- - -## Integration Points - -### Member Search Vector - -**Trigger Update:** -- When `member_groups` record created/deleted -- Update `members.search_vector` to include group names -- Use PostgreSQL trigger for automatic updates - -**Search Query:** -- Extend existing `fuzzy_search` to include group names -- Group names added with weight 'B' (same as city, etc.) - -### Member Form - -**Future Enhancement:** -- Add groups selection in member form -- Multi-select dropdown for groups -- Add/remove groups during member creation/edit - -### Authorization Integration - -**Current (MVP):** -- Only admins can manage groups -- Uses existing `Mv.Authorization.Checks.HasPermission` -- Permission: `groups` resource with `:all` scope - -**Future:** -- Group-specific permissions -- Role-based group management -- Member-level group assignment permissions - ---- +- **Do not use `role="status"` on group badges or navigation links.** That role is for live regions (screen-reader announcements), not for static labels or navigation. Use `aria-label` (e.g. "Member of group X") instead. +- `role="status"` with `aria-live="polite"` is appropriate only for dynamic announcements (filter changes, member-count updates). +- Clickable filter badges (optional enhancement) must be real `