Cleanup of the docs closes #507 #530
4 changed files with 436 additions and 1904 deletions
|
|
@ -1,796 +1,172 @@
|
|||
# CSV Member Import v1 - Implementation Plan
|
||||
# CSV Member Import
|
||||
|
||||
**Version:** 1.0
|
||||
**Last Updated:** 2026-01-13
|
||||
**Status:** In Progress (Backend Complete, UI Complete, Tests Pending)
|
||||
**Related Documents:**
|
||||
- [Feature Roadmap](./feature-roadmap.md) - Overall feature planning
|
||||
Reference for how the CSV member import actually behaves. The end-to-end
|
||||
LiveView test (`test/mv_web/live/import_live_test.exs`) and future maintenance
|
||||
depend on the rules documented here.
|
||||
|
||||
## Implementation Status
|
||||
**Status:** implemented (backend + LiveView UI).
|
||||
|
||||
**Completed Issues:**
|
||||
- ✅ Issue #1: CSV Specification & Static Template Files
|
||||
- ✅ Issue #2: Import Service Module Skeleton
|
||||
- ✅ Issue #3: CSV Parsing + Delimiter Auto-Detection + BOM Handling
|
||||
- ✅ Issue #4: Header Normalization + Per-Header Mapping
|
||||
- ✅ Issue #5: Validation (Required Fields) + Error Formatting
|
||||
- ✅ Issue #6: Persistence via Ash Create + Per-Row Error Capture (with Error-Capping)
|
||||
- ✅ Issue #7: Admin Global Settings LiveView UI (Upload + Start Import + Results + Template Links)
|
||||
- ✅ Issue #8: Authorization + Limits
|
||||
- ✅ Issue #11: Custom Field Import (Backend + UI)
|
||||
Implementation:
|
||||
|
||||
**In Progress / Pending:**
|
||||
- ⏳ Issue #9: End-to-End LiveView Tests + Fixtures
|
||||
- ⏳ Issue #10: Documentation Polish
|
||||
- `lib/mv/membership/import/csv_parser.ex` — BOM stripping, delimiter detection, physical line numbering
|
||||
- `lib/mv/membership/import/header_mapper.ex` — header normalization + column mapping
|
||||
- `lib/mv/membership/import/column_resolver.ex` — read-only resolution of groups + fee-type columns (preview)
|
||||
- `lib/mv/membership/import/member_csv.ex` — `prepare/2`, `process_chunk/4`, validation, member creation
|
||||
- `lib/mv/membership/import/import_runner.ex` — orchestration glue
|
||||
- `lib/mv_web/live/import_live.ex` (+ `import_live/components.ex`) — UI, state machine, chunk driving
|
||||
- `lib/mv_web/controllers/import_template_controller.ex` — on-the-fly template generation
|
||||
|
||||
**Latest Update:** CSV Import UI fully implemented in GlobalSettingsLive with chunk processing, progress tracking, error display, and custom field support (2026-01-13)
|
||||
## Scope
|
||||
|
||||
---
|
||||
Admin-only bulk creation of members from an uploaded CSV.
|
||||
|
||||
## Table of Contents
|
||||
- **Create only** — no upsert/update of existing members.
|
||||
- **No deduplication** — a duplicate email fails its row (unique constraint) and is reported as an error.
|
||||
- **Best-effort, row-by-row** — no transactional rollback; a failed row does not abort the import.
|
||||
- **No background jobs** — progress is driven via LiveView `handle_info` chunk messages.
|
||||
- **Errors shown in UI only** — no error-CSV export.
|
||||
|
||||
- [Overview & Scope](#overview--scope)
|
||||
- [UX Flow](#ux-flow)
|
||||
- [CSV Specification](#csv-specification)
|
||||
- [Technical Design Notes](#technical-design-notes)
|
||||
- [Implementation Issues](#implementation-issues)
|
||||
- [Rollout & Risks](#rollout--risks)
|
||||
Out of scope: upsert, mapping wizard, transactional all-or-nothing, error export, import history/audit.
|
||||
|
||||
---
|
||||
## UI Flow
|
||||
|
||||
## Overview & Scope
|
||||
- **Route:** `/admin/import` (LiveView `MvWeb.ImportLive`). Template downloads:
|
||||
`/admin/import/template/en` and `/admin/import/template/de` (dynamic controller, not static files).
|
||||
- **Authorization:** requires `can?(:create, Mv.Membership.Member)`. Non-admins are
|
||||
redirected with a "don't have permission" flash. The import section, the template
|
||||
controller, and the `start_import` event all enforce this.
|
||||
- **Upload:** `allow_upload(:csv_file, accept: .csv, max_entries: 1, auto_upload: true)`.
|
||||
File size limit enforced by `max_file_size`.
|
||||
- **State machine** (`@import_status`): `idle → preview → running → done|error`.
|
||||
- **start_import** parses + resolves the file and transitions to **preview**. This step
|
||||
is **read-only**: no members are created yet. The preview shows the column mapping,
|
||||
sample rows, groups that exist vs. would be created, and fee-type/unknown-column warnings.
|
||||
- **confirm_import** begins processing and creates members chunk by chunk.
|
||||
- **Results:** success count, failure count, error list (each with CSV line number, message,
|
||||
optional field), warnings, and a truncation notice when errors exceed the cap.
|
||||
|
||||
### What We're Building
|
||||
## Limits
|
||||
|
||||
A **basic CSV member import feature** that allows administrators to upload a CSV file and import new members into the system. This is a **v1 minimal implementation** focused on establishing the import structure without advanced features.
|
||||
- **Max file size:** configurable via `config :mv, csv_import: [max_file_size_mb: ...]` (enforced by `allow_upload`).
|
||||
- **Max rows:** configurable via `config :mv, csv_import: [max_rows: ...]`, default **1000**, excluding header. Enforced in `MemberCSV.prepare/2`; exceeding it yields an error containing `"exceeds"`.
|
||||
- **Chunk size:** 200 rows per chunk.
|
||||
- **Error cap:** 50 errors collected per import overall (`failed` count stays accurate; `errors_truncated?` flag set when exceeded).
|
||||
|
||||
**Core Functionality (v1 Minimal):**
|
||||
- Upload CSV file via LiveView file upload
|
||||
- Parse CSV with bilingual header support for core member fields (English/German)
|
||||
- Auto-detect delimiter (`;` or `,`) using header recognition
|
||||
- Map CSV columns to core member fields (`first_name`, `last_name`, `email`, `street`, `postal_code`, `city`, `country`)
|
||||
- **Import custom field values** - Map CSV columns to existing custom fields by name (unknown custom field columns will be ignored with a warning)
|
||||
- Validate each row (required field: `email`)
|
||||
- Create members via Ash resource (one-by-one, **no background jobs**, processed in chunks of 200 rows via LiveView messages)
|
||||
- Display import results: success count, error count, and error details
|
||||
- Provide static CSV templates (EN/DE)
|
||||
## Parsing (`CsvParser.parse/1`)
|
||||
|
||||
**Key Constraints (v1):**
|
||||
- ✅ **Admin-only feature**
|
||||
- ✅ **No upsert** (create only)
|
||||
- ✅ **No deduplication** (duplicate emails fail and show as errors)
|
||||
- ✅ **No mapping wizard** (fixed header mapping via bilingual variants)
|
||||
- ✅ **No background jobs** (progress via LiveView `handle_info`)
|
||||
- ✅ **Best-effort import** (row-by-row, no rollback)
|
||||
- ✅ **UI-only error display** (no error CSV export)
|
||||
- ✅ **Safety limits** (10 MB, 1,000 rows, chunks of 200)
|
||||
- Content must be **valid UTF-8** (else error). Empty content / empty header row are errors.
|
||||
- **UTF-8 BOM is stripped first**, before any header handling.
|
||||
- Line endings normalized: `\r\n`, `\r`, `\n` all handled.
|
||||
- **Delimiter auto-detection:** parse the header with both `;` and `,` parsers (NimbleCSV,
|
||||
quote-aware), count non-empty fields each yields, pick the higher; **`;` wins ties**;
|
||||
default `;`.
|
||||
- **Quoting:** double-quote quoting; `""` inside a quoted field is a literal `"`. Newlines
|
||||
inside quoted fields are supported — the record keeps its **start** line number.
|
||||
- **Physical line numbers:** rows are returned as `{csv_line_number, values}` where the line
|
||||
number is the physical 1-based line in the file (header is line 1, first data row is line 2).
|
||||
**Empty lines are skipped but do not shift numbering** — downstream code must use the
|
||||
parser's line numbers, never recompute from row index. (Test asserts an invalid row after a
|
||||
skipped empty line still reports its true physical line, e.g. `Line 4`.)
|
||||
- Completely empty rows are skipped. An unparsable row produces an error naming its line number.
|
||||
|
||||
### Out of Scope (v1)
|
||||
## Header Mapping & Normalization (`HeaderMapper`)
|
||||
|
||||
**Deferred to Future Versions:**
|
||||
- ❌ Upsert/update existing members
|
||||
- ❌ Advanced deduplication strategies
|
||||
- ❌ Column mapping wizard UI
|
||||
- ❌ Background job processing (Oban/GenStage)
|
||||
- ❌ Transactional all-or-nothing import
|
||||
- ❌ Error CSV export/download
|
||||
- ❌ Batch validation preview before import
|
||||
- ❌ Dynamic template generation
|
||||
- ❌ Import history/audit log
|
||||
- ❌ Import templates for other entities
|
||||
**`normalize_header/1`** (applied identically to incoming headers, mapping variants, custom
|
||||
field names, group names, and fee-type names):
|
||||
|
||||
---
|
||||
1. trim, lowercase
|
||||
2. transliterate German chars: `ß → ss`, `ä → ae`, `ö → oe`, `ü → ue` (and uppercase forms)
|
||||
3. unify hyphen variants (en dash U+2013, em dash U+2014, minus U+2212 → `-`)
|
||||
4. punctuation to spaces: `_`, `()[]{}`, `/`, `\` → space
|
||||
5. **remove all whitespace** (so `first name` == `firstname`)
|
||||
6. final trim
|
||||
|
||||
## UX Flow
|
||||
Matching is on the fully normalized string.
|
||||
|
||||
### Access & Location
|
||||
**Required field:** `email`. Missing it aborts `prepare` with a "Missing required header" error.
|
||||
|
||||
**Entry Point:**
|
||||
- **Location:** Global Settings page (`/settings`)
|
||||
- **UI Element:** New section "Import Members (CSV)" below "Custom Fields" section
|
||||
- **Access Control:** Admin-only (enforced at LiveView event level, not entire `/settings` route)
|
||||
**Unknown member-field columns:** ignored (no error). If an unknown column looks like it
|
||||
could be a custom field that does not exist, a **warning** is emitted (import continues).
|
||||
|
||||
### User Journey
|
||||
**Duplicate headers** mapping to the same canonical field (or same custom field) are an error.
|
||||
|
||||
1. **Navigate to Global Settings**
|
||||
2. **Access Import Section**
|
||||
- **Important notice:** Custom fields should be created in Mila before importing CSV files with custom field columns (unknown columns will be ignored with a warning)
|
||||
- Upload area (drag & drop or file picker)
|
||||
- Template download links (English / German)
|
||||
- Help text explaining CSV format and custom field requirements
|
||||
3. **Ensure Custom Fields Exist (if importing custom fields)**
|
||||
- Navigate to Custom Fields section and create required custom fields
|
||||
- Note the name/identifier for each custom field (used as CSV header)
|
||||
4. **Download Template (Optional)**
|
||||
5. **Prepare CSV File**
|
||||
- Include custom field columns using the custom field name as header (e.g., `membership_number`, `birth_date`)
|
||||
6. **Upload CSV**
|
||||
7. **Start Import**
|
||||
- Runs server-side via LiveView messages (may take up to ~30 seconds for large files)
|
||||
- Warning messages if custom field columns reference non-existent custom fields (columns will be ignored)
|
||||
8. **View Results**
|
||||
- Success count
|
||||
- Error count
|
||||
- First 50 errors, each with:
|
||||
- **CSV line number** (header is line 1, first data record begins at line 2)
|
||||
- Error message
|
||||
- Field name (if applicable)
|
||||
### Supported member fields and header variants
|
||||
|
||||
### Error Handling
|
||||
Source of truth is `@member_field_variants_raw` in `header_mapper.ex`. Variants below are
|
||||
illustrative; matching is via normalization, so casing/hyphen/whitespace differences all collapse.
|
||||
|
||||
- **File too large:** Flash error before upload starts
|
||||
- **Too many rows:** Flash error before import starts
|
||||
- **Invalid CSV format:** Error shown in results
|
||||
- **Partial success:** Results show both success and error counts
|
||||
|
||||
---
|
||||
|
||||
## CSV Specification
|
||||
|
||||
### Delimiter
|
||||
|
||||
**Recommended:** Semicolon (`;`)
|
||||
**Supported:** `;` and `,`
|
||||
|
||||
**Auto-Detection (Header Recognition):**
|
||||
- Remove UTF-8 BOM *first*
|
||||
- Extract header record and try parsing with both delimiters
|
||||
- For each delimiter, count how many recognized headers are present (via normalized variants)
|
||||
- Choose delimiter with higher recognition; prefer `;` if tied
|
||||
- If neither yields recognized headers, default to `;`
|
||||
|
||||
### Quoting Rules
|
||||
|
||||
- Fields may be quoted with double quotes (`"`)
|
||||
- Escaped quotes: `""` inside quoted field represents a single `"`
|
||||
- **v1 assumption:** CSV records do **not** contain embedded newlines inside quoted fields. (If they do, parsing may fail or line numbers may be inaccurate.)
|
||||
|
||||
### Column Headers
|
||||
|
||||
**v1 Supported Fields:**
|
||||
|
||||
**Core Member Fields (all importable):**
|
||||
- `email` / `E-Mail` (required)
|
||||
- `first_name` / `Vorname` (optional)
|
||||
- `last_name` / `Nachname` (optional)
|
||||
- `join_date` / `Beitrittsdatum` (optional, ISO-8601 date)
|
||||
- `exit_date` / `Austrittsdatum` (optional, ISO-8601 date)
|
||||
- `notes` / `Notizen` (optional)
|
||||
- `country` / `Land` / `Staat` (optional)
|
||||
- `city` / `Stadt` (optional)
|
||||
- `street` / `Straße` (optional)
|
||||
- `house_number` / `Hausnummer` / `Nr.` (optional)
|
||||
- `postal_code` / `PLZ` / `Postleitzahl` (optional)
|
||||
- `membership_fee_start_date` / `Beitragsbeginn` (optional, ISO-8601 date)
|
||||
|
||||
Address column order in import/export matches the members overview: country, city, street, house number, postal code.
|
||||
|
||||
**Not supported for import (by design):**
|
||||
- **membership_fee_status** – Computed field (from fee cycles). Not stored; export-only.
|
||||
- **groups** – Many-to-many relationship. Would require resolving group names to IDs; not in current scope.
|
||||
- **membership_fee_type_id** – Foreign key; could be added later (e.g. resolve type name to ID).
|
||||
|
||||
**Custom Fields:**
|
||||
- Any custom field column using the custom field's **name** as the header (e.g., `membership_number`, `birth_date`)
|
||||
- **Important:** Custom fields must be created in Mila before importing. The CSV header must match the custom field name exactly (same normalization as member fields).
|
||||
- **Behavior:** If the CSV contains custom field columns that don't exist in Mila, a warning message will be shown and those columns will be ignored during import.
|
||||
- **Value Validation:** Custom field values are validated according to the custom field type:
|
||||
- **string**: Any text value (trimmed)
|
||||
- **integer**: Must be a valid integer (e.g., `42`, `-10`). Invalid values will cause a row error with the custom field name and reason.
|
||||
- **boolean**: Accepts `true`, `false`, `1`, `0`, `yes`, `no`, `ja`, `nein` (case-insensitive). Invalid values will cause a row error.
|
||||
- **date**: Must be in ISO-8601 format (YYYY-MM-DD, e.g., `2024-01-15`). Invalid values will cause a row error.
|
||||
- **email**: Must be a valid email format (contains `@`, 5-254 characters, valid format). Invalid values will cause a row error.
|
||||
- **Error Messages:** Custom field validation errors are included in the import error list with format: `custom_field: <name> – <reason>` (e.g., `custom_field: Alter – expected integer, got: abc`)
|
||||
|
||||
**Member Field Header Mapping:**
|
||||
|
||||
| Canonical Field | English Variants | German Variants |
|
||||
| Canonical | Example accepted headers (EN / DE) | Notes |
|
||||
|---|---|---|
|
||||
| `first_name` | `first_name`, `firstname` | `Vorname`, `vorname` |
|
||||
| `last_name` | `last_name`, `lastname`, `surname` | `Nachname`, `nachname`, `Familienname` |
|
||||
| `email` | `email`, `e-mail`, `e_mail` | `E-Mail`, `e-mail`, `e_mail` |
|
||||
| `join_date` | `join date`, `join_date` | `Beitrittsdatum`, `beitritts-datum` |
|
||||
| `exit_date` | `exit date`, `exit_date` | `Austrittsdatum`, `austritts-datum` |
|
||||
| `notes` | `notes` | `Notizen`, `bemerkungen` |
|
||||
| `street` | `street`, `address` | `Straße`, `strasse`, `Strasse` |
|
||||
| `house_number` | `house number`, `house_number`, `house no` | `Hausnummer`, `Nr`, `Nr.`, `Nummer` |
|
||||
| `postal_code` | `postal_code`, `zip`, `postcode` | `PLZ`, `plz`, `Postleitzahl`, `postleitzahl` |
|
||||
| `city` | `city`, `town` | `Stadt`, `stadt`, `Ort` |
|
||||
| `country` | `country` | `Land`, `land`, `Staat`, `staat` |
|
||||
| `membership_fee_start_date` | `membership fee start date`, `membership_fee_start_date`, `fee start` | `Beitragsbeginn`, `beitrags-beginn` |
|
||||
|
||||
**Header Normalization (used consistently for both input headers AND mapping variants):**
|
||||
- Trim whitespace
|
||||
- Convert to lowercase
|
||||
- Normalize Unicode: `ß` → `ss` (e.g., `Straße` → `strasse`)
|
||||
- Replace hyphens/whitespace with underscores: `E-Mail` → `e_mail`, `phone number` → `phone_number`
|
||||
- Collapse multiple underscores: `e__mail` → `e_mail`
|
||||
- Case-insensitive matching
|
||||
|
||||
**Unknown columns:** ignored (no error)
|
||||
|
||||
**Required fields:** `email`
|
||||
|
||||
**Custom Field Columns:**
|
||||
- Custom field columns are identified by matching the normalized CSV header to the custom field `name` (not slug)
|
||||
- Same normalization rules apply as for member fields (trim, lowercase, Unicode normalization, underscore replacement)
|
||||
- Unknown custom field columns (non-existent names) will be ignored with a warning message
|
||||
|
||||
### CSV Template Files
|
||||
|
||||
**Location:**
|
||||
- `priv/static/templates/member_import_en.csv`
|
||||
- `priv/static/templates/member_import_de.csv`
|
||||
|
||||
**Content:**
|
||||
- Header row with required + common optional fields
|
||||
- **Note:** Custom field columns are not included in templates by default (users add them based on their custom field configuration)
|
||||
- One example row
|
||||
- Uses semicolon delimiter (`;`)
|
||||
- UTF-8 encoding **with BOM** (Excel compatibility)
|
||||
|
||||
**Template Access:**
|
||||
- Templates are static files in `priv/static/templates/`
|
||||
- Served at:
|
||||
- `/templates/member_import_en.csv`
|
||||
- `/templates/member_import_de.csv`
|
||||
- In LiveView, link them using Phoenix static path helpers (e.g. `~p` or `Routes.static_path/2`, depending on Phoenix version).
|
||||
|
||||
**Example Usage in LiveView Templates:**
|
||||
|
||||
```heex
|
||||
<!-- Using ~p sigil (Phoenix 1.7+) -->
|
||||
<.link href={~p"/templates/member_import_en.csv"} download>
|
||||
<%= gettext("Download English Template") %>
|
||||
</.link>
|
||||
|
||||
<.link href={~p"/templates/member_import_de.csv"} download>
|
||||
<%= gettext("Download German Template") %>
|
||||
</.link>
|
||||
|
||||
<!-- Alternative: Using Routes.static_path/2 -->
|
||||
<.link href={Routes.static_path(MvWeb.Endpoint, "/templates/member_import_en.csv")} download>
|
||||
<%= gettext("Download English Template") %>
|
||||
</.link>
|
||||
```
|
||||
|
||||
**Note:** The `templates` directory must be included in `MvWeb.static_paths()` (configured in `lib/mv_web.ex`) for the files to be served.
|
||||
|
||||
### File Limits
|
||||
|
||||
- **Max file size:** 10 MB
|
||||
- **Max rows:** 1,000 rows (excluding header)
|
||||
- **Processing:** chunks of 200 (via LiveView messages)
|
||||
- **Encoding:** UTF-8 (BOM handled)
|
||||
|
||||
---
|
||||
|
||||
## Technical Design Notes
|
||||
|
||||
### Architecture Overview
|
||||
|
||||
```
|
||||
┌─────────────────┐
|
||||
│ LiveView UI │ (GlobalSettingsLive or component)
|
||||
│ - Upload area │
|
||||
│ - Progress │
|
||||
│ - Results │
|
||||
└────────┬────────┘
|
||||
│ prepare
|
||||
▼
|
||||
┌─────────────────────────────┐
|
||||
│ Import Service │ (Mv.Membership.Import.MemberCSV)
|
||||
│ - parse + map + limit checks│ -> returns import_state
|
||||
│ - process_chunk(chunk) │ -> returns chunk results
|
||||
└────────┬────────────────────┘
|
||||
│ create
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Ash Resource │ (Mv.Membership.Member)
|
||||
│ - Create │
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
### Technology Stack
|
||||
|
||||
- **Phoenix LiveView:** file upload via `allow_upload/3`
|
||||
- **NimbleCSV:** CSV parsing (add explicit dependency if missing)
|
||||
- **Ash Resource:** member creation via `Membership.create_member/1`
|
||||
- **Gettext:** bilingual UI/error messages
|
||||
|
||||
### Module Structure
|
||||
|
||||
**New Modules:**
|
||||
- `lib/mv/membership/import/member_csv.ex` - import orchestration + chunk processing + custom field handling
|
||||
- `lib/mv/membership/import/csv_parser.ex` - delimiter detection + parsing + BOM handling
|
||||
- `lib/mv/membership/import/header_mapper.ex` - normalization + header mapping (core fields + custom fields)
|
||||
|
||||
**Modified Modules:**
|
||||
- `lib/mv_web/live/global_settings_live.ex` - render import section, handle upload/events/messages
|
||||
|
||||
### Data Flow
|
||||
|
||||
1. **Upload:** LiveView receives file via `allow_upload`
|
||||
2. **Consume:** `consume_uploaded_entries/3` reads file content
|
||||
3. **Prepare:** `MemberCSV.prepare/2`
|
||||
- Strip BOM
|
||||
- Detect delimiter (header recognition)
|
||||
- Parse header + rows
|
||||
- Map headers to canonical fields (core member fields)
|
||||
- **Query existing custom fields and map custom field columns by name** (using same normalization as member fields)
|
||||
- **Warn about unknown custom field columns** (non-existent names will be ignored with warning)
|
||||
- Early abort if required headers missing
|
||||
- Row count check
|
||||
- Return `import_state` containing chunks, column_map, and custom_field_map
|
||||
4. **Process:** LiveView drives chunk processing via `handle_info`
|
||||
- For each chunk: validate + create member + create custom field values + collect errors
|
||||
5. **Results:** LiveView shows progress + final summary
|
||||
|
||||
### Types & Key Consistency
|
||||
|
||||
- **Raw CSV parsing:** returns headers as list of strings, and rows **with csv line numbers**
|
||||
- **Header mapping:** operates on normalized strings; mapping table variants are normalized once
|
||||
- **Ash attrs:** built as atom-keyed map (`%{first_name: ..., ...}`)
|
||||
|
||||
### Error Model
|
||||
|
||||
```elixir
|
||||
%{
|
||||
csv_line_number: 5, # physical line number in the CSV file
|
||||
field: :email, # optional
|
||||
message: "is not a valid email"
|
||||
}
|
||||
```
|
||||
|
||||
### CSV Line Numbers (Important)
|
||||
|
||||
To keep error reporting user-friendly and accurate, **row errors must reference the physical line number in the original file**, even if empty lines are skipped.
|
||||
|
||||
**Design decision:** the parser returns rows as:
|
||||
|
||||
```elixir
|
||||
rows :: [{csv_line_number :: pos_integer(), row_map :: map()}]
|
||||
```
|
||||
|
||||
Downstream logic must **not** recompute line numbers from row indexes.
|
||||
|
||||
### Authorization
|
||||
|
||||
**Enforcement points:**
|
||||
1. **LiveView event level:** check admin permission in `handle_event("start_import", ...)`
|
||||
2. **UI level:** render import section only for admin users
|
||||
3. **Static templates:** public assets (no authorization needed)
|
||||
|
||||
Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string checks where possible.
|
||||
|
||||
### Safety Limits
|
||||
|
||||
- File size enforced by `allow_upload` (`max_file_size`)
|
||||
- Row count enforced in `MemberCSV.prepare/2` before processing starts
|
||||
- Chunking is done via **LiveView `handle_info` loop** (sequential, cooperative scheduling)
|
||||
|
||||
---
|
||||
|
||||
## Implementation Issues
|
||||
|
||||
### Issue #1: CSV Specification & Static Template Files
|
||||
|
||||
**Dependencies:** None
|
||||
|
||||
**Status:** ✅ **COMPLETED**
|
||||
|
||||
**Goal:** Define CSV contract and add static templates.
|
||||
|
||||
**Tasks:**
|
||||
- [x] Finalize header mapping variants
|
||||
- [x] Document normalization rules
|
||||
- [x] Document delimiter detection strategy
|
||||
- [x] Create templates in `priv/static/templates/` (UTF-8 with BOM)
|
||||
- `member_import_en.csv` with English headers
|
||||
- `member_import_de.csv` with German headers
|
||||
- [x] Document template URLs and how to link them from LiveView
|
||||
- [x] Document line number semantics (physical CSV line numbers)
|
||||
- [x] Templates included in `MvWeb.static_paths()` configuration
|
||||
|
||||
**Definition of Done:**
|
||||
- [x] Templates open cleanly in Excel/LibreOffice
|
||||
- [x] CSV spec section complete
|
||||
|
||||
---
|
||||
|
||||
### Issue #2: Import Service Module Skeleton
|
||||
|
||||
**Dependencies:** None
|
||||
|
||||
**Status:** ✅ **COMPLETED**
|
||||
|
||||
**Goal:** Create service API and error types.
|
||||
|
||||
**API (recommended):**
|
||||
- `prepare/2` — parse + map + limit checks, returns import_state
|
||||
- `process_chunk/4` — process one chunk (pure-ish), returns per-chunk results
|
||||
|
||||
**Tasks:**
|
||||
- [x] Create `lib/mv/membership/import/member_csv.ex`
|
||||
- [x] Define public function: `prepare/2 (file_content, opts \\ [])`
|
||||
- [x] Define public function: `process_chunk/4 (chunk_rows_with_lines, column_map, custom_field_map, opts \\ [])`
|
||||
- [x] Define error struct: `%MemberCSV.Error{csv_line_number: integer, field: atom | nil, message: String.t}`
|
||||
- [x] Document module + API
|
||||
|
||||
---
|
||||
|
||||
### Issue #3: CSV Parsing + Delimiter Auto-Detection + BOM Handling
|
||||
|
||||
**Dependencies:** Issue #2
|
||||
|
||||
**Status:** ✅ **COMPLETED**
|
||||
|
||||
**Goal:** Parse CSV robustly with correct delimiter detection and BOM handling.
|
||||
|
||||
**Tasks:**
|
||||
- [x] Verify/add NimbleCSV dependency (`{:nimble_csv, "~> 1.0"}`)
|
||||
- [x] Create `lib/mv/membership/import/csv_parser.ex`
|
||||
- [x] Implement `strip_bom/1` and apply it **before** any header handling
|
||||
- [x] Handle `\r\n` and `\n` line endings (trim `\r` on header record)
|
||||
- [x] Detect delimiter via header recognition (try `;` and `,`)
|
||||
- [x] Parse CSV and return:
|
||||
- `headers :: [String.t()]`
|
||||
- `rows :: [{csv_line_number, [String.t()]}]` with correct physical line numbers
|
||||
- [x] Skip completely empty records (but preserve correct physical line numbers)
|
||||
- [x] Return `{:ok, headers, rows}` or `{:error, reason}`
|
||||
|
||||
**Definition of Done:**
|
||||
- [x] BOM handling works (Excel exports)
|
||||
- [x] Delimiter detection works reliably
|
||||
- [x] Rows carry correct `csv_line_number`
|
||||
|
||||
---
|
||||
|
||||
### Issue #4: Header Normalization + Per-Header Mapping (No Language Detection)
|
||||
|
||||
**Dependencies:** Issue #3
|
||||
|
||||
**Status:** ✅ **COMPLETED**
|
||||
|
||||
**Goal:** Map each header individually to canonical fields (normalized comparison).
|
||||
|
||||
**Tasks:**
|
||||
- [x] Create `lib/mv/membership/import/header_mapper.ex`
|
||||
- [x] Implement `normalize_header/1`
|
||||
- [x] Normalize mapping variants once and compare normalized strings
|
||||
- [x] Build `column_map` (canonical field -> column index)
|
||||
- [x] **Early abort if required headers missing** (`email`)
|
||||
- [x] Ignore unknown columns (member fields only)
|
||||
- [x] **Separate custom field column detection** (by name, with normalization)
|
||||
|
||||
**Definition of Done:**
|
||||
- [x] English/German headers map correctly
|
||||
- [x] Missing required columns fails fast
|
||||
|
||||
---
|
||||
|
||||
### Issue #5: Validation (Required Fields) + Error Formatting
|
||||
|
||||
**Dependencies:** Issue #4
|
||||
|
||||
**Status:** ✅ **COMPLETED**
|
||||
|
||||
**Goal:** Validate each row and return structured, translatable errors.
|
||||
|
||||
**Tasks:**
|
||||
- [x] Implement `validate_row/3 (row_map, csv_line_number, opts)`
|
||||
- [x] Required field presence (`email`)
|
||||
- [x] Email format validation (EctoCommons.EmailValidator)
|
||||
- [x] Trim values before validation
|
||||
- [x] Gettext-backed error messages
|
||||
|
||||
---
|
||||
|
||||
### Issue #6: Persistence via Ash Create + Per-Row Error Capture (Chunked Processing)
|
||||
|
||||
**Dependencies:** Issue #5
|
||||
|
||||
**Status:** ✅ **COMPLETED**
|
||||
|
||||
**Goal:** Create members and capture errors per row with correct CSV line numbers.
|
||||
|
||||
**Tasks:**
|
||||
- [x] Implement `process_chunk/4` in service:
|
||||
- Input: `[{csv_line_number, row_map}]`
|
||||
- Validate + create sequentially
|
||||
- Collect counts + first 50 errors (per import overall; LiveView enforces cap across chunks)
|
||||
- **Error-Capping:** Supports `existing_error_count` and `max_errors` in opts (default: 50)
|
||||
- **Error-Capping:** Only collects errors if under limit, but continues processing all rows
|
||||
- **Error-Capping:** `failed` count is always accurate, even when errors are capped
|
||||
- [x] Implement Ash error formatter helper:
|
||||
- Convert `Ash.Error.Invalid` into `%MemberCSV.Error{}`
|
||||
- Prefer field-level errors where possible (attach `field` atom)
|
||||
- Handle unique email constraint error as user-friendly message
|
||||
- [x] Map row_map to Ash attrs (`%{first_name: ..., ...}`)
|
||||
- [x] Custom field value processing and creation
|
||||
|
||||
**Important:** **Do not recompute line numbers** in this layer—use the ones provided by the parser.
|
||||
|
||||
**Implementation Notes:**
|
||||
- `process_chunk/4` accepts `opts` with `existing_error_count` and `max_errors` for error capping across chunks
|
||||
- Error capping respects the limit per import overall (not per chunk)
|
||||
- Processing continues even after error limit is reached (for accurate counts)
|
||||
|
||||
---
|
||||
|
||||
### Issue #7: Admin Global Settings LiveView UI (Upload + Start Import + Results + Template Links)
|
||||
|
||||
**Dependencies:** Issue #6
|
||||
|
||||
**Status:** ✅ **COMPLETED**
|
||||
|
||||
**Goal:** UI section with upload, progress, results, and template links.
|
||||
|
||||
**Tasks:**
|
||||
- [x] Render import section only for admins
|
||||
- [x] **Add prominent UI notice about custom fields:**
|
||||
- Display alert/info box: "Custom fields must be created in Mila before importing CSV files with custom field columns"
|
||||
- Explain: "Use the custom field name as the CSV column header (same normalization as member fields applies)"
|
||||
- Add link to custom fields management section
|
||||
- [x] Configure `allow_upload/3`:
|
||||
- `.csv` only, `max_entries: 1`, `max_file_size: 10MB`, `auto_upload: true` (auto-upload enabled for better UX)
|
||||
- [x] `handle_event("start_import", ...)`:
|
||||
- Admin permission check
|
||||
- Consume upload -> read file content
|
||||
- Call `MemberCSV.prepare/2`
|
||||
- Store `import_state` in assigns (chunks + column_map + metadata)
|
||||
- Initialize progress assigns
|
||||
- `send(self(), {:process_chunk, 0})`
|
||||
- [x] `handle_info({:process_chunk, idx}, socket)`:
|
||||
- Fetch chunk from `import_state`
|
||||
- Call `MemberCSV.process_chunk/4` with error capping support
|
||||
- Merge counts/errors into progress assigns (cap errors at 50 overall)
|
||||
- Schedule next chunk (or finish and show results)
|
||||
- Async task processing with SQL sandbox support for tests
|
||||
- [x] Results UI:
|
||||
- Success count
|
||||
- Failure count
|
||||
- Error list (line number + message + field)
|
||||
- **Warning messages for unknown custom field columns** (non-existent names) shown in results
|
||||
- Progress indicator during import
|
||||
- Error truncation notice when errors exceed limit
|
||||
|
||||
**Template links:**
|
||||
- [x] Link `/templates/member_import_en.csv` and `/templates/member_import_de.csv` via Phoenix static path helpers.
|
||||
|
||||
**Definition of Done:**
|
||||
- [x] Upload area with drag & drop support
|
||||
- [x] Template download links (EN/DE)
|
||||
- [x] Progress tracking during import
|
||||
- [x] Results display with success/error counts
|
||||
- [x] Error list with line numbers and field information
|
||||
- [x] Warning display for unknown custom field columns
|
||||
- [x] Admin-only access control
|
||||
- [x] Async chunk processing with proper error handling
|
||||
|
||||
---
|
||||
|
||||
### Issue #8: Authorization + Limits
|
||||
|
||||
**Dependencies:** None (can be parallelized)
|
||||
|
||||
**Status:** ✅ **COMPLETED**
|
||||
|
||||
**Goal:** Ensure admin-only access and enforce limits.
|
||||
|
||||
**Tasks:**
|
||||
- [x] Admin check in start import event handler (via `Authorization.can?/3`)
|
||||
- [x] File size enforced in upload config (`max_file_size: 10MB`)
|
||||
- [x] Row limit enforced in `MemberCSV.prepare/2` (max_rows: 1000, configurable via opts)
|
||||
- [x] Chunk size limit (200 rows per chunk)
|
||||
- [x] Error limit (50 errors per import)
|
||||
- [x] UI-level authorization check (import section only visible to admins)
|
||||
- [x] Event-level authorization check (prevents unauthorized import attempts)
|
||||
|
||||
**Implementation Notes:**
|
||||
- File size limit: 10 MB (10,485,760 bytes) enforced via `allow_upload/3`
|
||||
- Row limit: 1,000 rows (excluding header) enforced in `MemberCSV.prepare/2`
|
||||
- Chunk size: 200 rows per chunk (configurable via opts)
|
||||
- Error limit: 50 errors per import (configurable via `@max_errors`)
|
||||
- Authorization uses `MvWeb.Authorization.can?/3` with `:create` permission on `Mv.Membership.Member`
|
||||
|
||||
**Definition of Done:**
|
||||
- [x] Admin-only access enforced at UI and event level
|
||||
- [x] File size limit enforced
|
||||
- [x] Row count limit enforced
|
||||
- [x] Chunk processing with size limits
|
||||
- [x] Error capping implemented
|
||||
|
||||
---
|
||||
|
||||
### Issue #9: End-to-End LiveView Tests + Fixtures
|
||||
|
||||
**Dependencies:** Issue #7 and #8
|
||||
|
||||
**Tasks:**
|
||||
- [ ] Fixtures:
|
||||
- valid EN/DE (core fields only)
|
||||
- valid with custom fields
|
||||
- invalid
|
||||
- unknown custom field name (non-existent, should show warning)
|
||||
- too many rows (1,001)
|
||||
- BOM + `;` delimiter fixture
|
||||
- fixture with empty line(s) to validate correct line numbers
|
||||
- [ ] LiveView tests:
|
||||
- admin sees section, non-admin does not
|
||||
- upload + start import
|
||||
- success + error rendering
|
||||
- row limit + file size errors
|
||||
- custom field import success
|
||||
- custom field import warning (non-existent name, column ignored)
|
||||
|
||||
---
|
||||
|
||||
### Issue #10: Documentation Polish (Inline Help Text + Docs)
|
||||
|
||||
**Dependencies:** Issue #9
|
||||
|
||||
**Tasks:**
|
||||
- [ ] UI help text + translations
|
||||
- [ ] CHANGELOG entry
|
||||
- [ ] Ensure moduledocs/docs
|
||||
|
||||
---
|
||||
|
||||
### Issue #11: Custom Field Import
|
||||
|
||||
**Dependencies:** Issue #6 (Persistence)
|
||||
|
||||
**Priority:** High (Core v1 Feature)
|
||||
|
||||
**Status:** ✅ **COMPLETED** (Backend + UI Implementation)
|
||||
|
||||
**Goal:** Support importing custom field values from CSV columns. Custom fields should exist in Mila before import for best results.
|
||||
|
||||
**Important Requirements:**
|
||||
- **Custom fields should be created in Mila first** - Unknown custom field columns will be ignored with a warning message
|
||||
- CSV headers for custom fields must match the custom field **name** exactly (same normalization as member fields applies)
|
||||
- Custom field values are validated according to the custom field type (string, integer, boolean, date, email)
|
||||
- Unknown custom field columns (non-existent names) will be ignored with a warning - import continues
|
||||
|
||||
**Tasks:**
|
||||
- [x] Extend `header_mapper.ex` to detect custom field columns by name (using same normalization as member fields)
|
||||
- [x] Query existing custom fields during `prepare/2` to map custom field columns
|
||||
- [x] Collect unknown custom field columns and add warning messages (don't fail import)
|
||||
- [x] Map custom field CSV values to `CustomFieldValue` creation in `process_chunk/4`
|
||||
- [x] Handle custom field type validation (string, integer, boolean, date, email) with proper error messages
|
||||
- [x] Create `CustomFieldValue` records linked to members during import
|
||||
- [x] Validate custom field values and return structured errors with custom field name and reason
|
||||
- [x] UI help text and link to custom field management (implemented in Issue #7)
|
||||
- [x] Update error messages to include custom field validation errors (format: `custom_field: <name> – expected <type>, got: <value>`)
|
||||
- [x] Add UI help text explaining custom field requirements (completed in Issue #7):
|
||||
- "Custom fields must be created in Mila before importing"
|
||||
- "Use the custom field name as the CSV column header (same normalization as member fields)"
|
||||
- Link to custom fields management section
|
||||
- [x] Update CSV templates documentation to explain custom field columns (documented in Issue #1)
|
||||
- [x] Add tests for custom field import (valid, invalid name, type validation, warning for unknown)
|
||||
|
||||
**Definition of Done:**
|
||||
- [x] Custom field columns are recognized by name (with normalization)
|
||||
- [x] Warning messages shown for unknown custom field columns (import continues)
|
||||
- [x] Custom field values are created and linked to members
|
||||
- [x] Type validation works for all custom field types (string, integer, boolean, date, email)
|
||||
- [x] UI clearly explains custom field requirements (completed in Issue #7)
|
||||
- [x] Tests cover custom field import scenarios (including warning for unknown names)
|
||||
- [x] Error messages include custom field validation errors with proper formatting
|
||||
|
||||
**Implementation Notes:**
|
||||
- Custom field lookup is built in `prepare/2` and passed via `custom_field_lookup` in opts
|
||||
- Custom field values are formatted according to type in `format_custom_field_value/2`
|
||||
- Unknown custom field columns generate warnings in `import_state.warnings`
|
||||
|
||||
---
|
||||
|
||||
## Rollout & Risks
|
||||
|
||||
### Rollout Strategy
|
||||
- Dev → Staging → Production (with anonymized real-world CSV tests)
|
||||
|
||||
### Risks & Mitigations
|
||||
|
||||
| Risk | Impact | Likelihood | Mitigation |
|
||||
|---|---:|---:|---|
|
||||
| Large import timeout | High | Medium | 10 MB + 1,000 rows, chunking via `handle_info` |
|
||||
| Encoding issues | Medium | Medium | BOM stripping, templates with BOM |
|
||||
| Invalid CSV format | Medium | High | Clear errors + templates |
|
||||
| Duplicate emails | Low | High | Ash constraint error -> user-friendly message |
|
||||
| Performance (no background jobs) | Medium | Low | Small limits, sequential chunk processing |
|
||||
| Admin access bypass | High | Low | Event-level auth + UI hiding |
|
||||
| Data corruption | High | Low | Per-row validation + best-effort |
|
||||
|
||||
---
|
||||
|
||||
## Appendix
|
||||
|
||||
### Module File Structure
|
||||
|
||||
```
|
||||
lib/
|
||||
├── mv/
|
||||
│ └── membership/
|
||||
│ └── import/
|
||||
│ ├── member_csv.ex # prepare + process_chunk
|
||||
│ ├── import_runner.ex # orchestration: file read, progress merge, chunk process, error format
|
||||
│ ├── csv_parser.ex # delimiter detection + parsing + BOM handling
|
||||
│ └── header_mapper.ex # normalization + header mapping
|
||||
└── mv_web/
|
||||
└── live/
|
||||
├── import_export_live.ex # mount / handle_event / handle_info + glue only
|
||||
└── import_export_live/
|
||||
└── components.ex # UI: custom_fields_notice, template_links, import_form, import_progress, import_results
|
||||
|
||||
priv/
|
||||
└── static/
|
||||
└── templates/
|
||||
├── member_import_en.csv
|
||||
└── member_import_de.csv
|
||||
|
||||
test/
|
||||
├── mv/
|
||||
│ └── membership/
|
||||
│ └── import/
|
||||
│ ├── member_csv_test.exs
|
||||
│ ├── csv_parser_test.exs
|
||||
│ └── header_mapper_test.exs
|
||||
└── fixtures/
|
||||
├── member_import_en.csv
|
||||
├── member_import_de.csv
|
||||
├── member_import_invalid.csv
|
||||
├── member_import_large.csv
|
||||
└── member_import_empty_lines.csv
|
||||
```
|
||||
|
||||
### Example Usage (LiveView)
|
||||
|
||||
```elixir
|
||||
def handle_event("start_import", _params, socket) do
|
||||
assert_admin!(socket.assigns.current_user)
|
||||
|
||||
[{_name, content}] =
|
||||
consume_uploaded_entries(socket, :csv_file, fn %{path: path}, _entry ->
|
||||
{:ok, File.read!(path)}
|
||||
end)
|
||||
|
||||
case Mv.Membership.Import.MemberCSV.prepare(content) do
|
||||
{:ok, import_state} ->
|
||||
socket =
|
||||
socket
|
||||
|> assign(:import_state, import_state)
|
||||
|> assign(:import_progress, %{processed: 0, inserted: 0, failed: 0, errors: []})
|
||||
|> assign(:importing?, true)
|
||||
|
||||
send(self(), {:process_chunk, 0})
|
||||
{:noreply, socket}
|
||||
|
||||
{:error, reason} ->
|
||||
{:noreply, put_flash(socket, :error, reason)}
|
||||
end
|
||||
end
|
||||
|
||||
def handle_info({:process_chunk, idx}, socket) do
|
||||
%{chunks: chunks, column_map: column_map} = socket.assigns.import_state
|
||||
|
||||
case Enum.at(chunks, idx) do
|
||||
nil ->
|
||||
{:noreply, assign(socket, importing?: false)}
|
||||
|
||||
chunk_rows_with_lines ->
|
||||
{:ok, chunk_result} =
|
||||
Mv.Membership.Import.MemberCSV.process_chunk(chunk_rows_with_lines, column_map)
|
||||
|
||||
socket = merge_progress(socket, chunk_result) # caps errors at 50 overall
|
||||
|
||||
send(self(), {:process_chunk, idx + 1})
|
||||
{:noreply, socket}
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**End of Implementation Plan**
|
||||
| `email` (required) | email, e-mail, e_mail, mail, e-mail-adresse / E-Mail | |
|
||||
| `first_name` | first name, firstname / Vorname | |
|
||||
| `last_name` | last name, lastname, surname / Nachname, Familienname | |
|
||||
| `join_date` | join date / Beitrittsdatum | ISO-8601 date |
|
||||
| `exit_date` | exit date / Austrittsdatum | ISO-8601 date |
|
||||
| `notes` | notes / Notizen, Bemerkungen | |
|
||||
| `street` | street, address / Straße, Strasse | |
|
||||
| `house_number` | house number, house no / Hausnummer, Nr, Nr., Nummer | |
|
||||
| `postal_code` | postal code, zip, postcode / PLZ, Postleitzahl | |
|
||||
| `city` | city, town / Stadt, Ort | |
|
||||
| `country` | country / Land, Staat | |
|
||||
| `membership_fee_start_date` | membership fee start date, fee start / Beitragsbeginn | ISO-8601 date |
|
||||
|
||||
### Special relationship columns
|
||||
|
||||
- **groups** (headers `Groups` / `Gruppen` / `Gruppe`) — comma-separated group names. Names
|
||||
matched case-insensitively against existing groups; **missing groups are auto-created** during
|
||||
processing. A group-assignment failure fails that row (the member was already created).
|
||||
- **membership_fee_type** (headers `Fee Type`, `fee_type`, `membership_fee_type` / `Beitragsart`)
|
||||
— name matched to an existing `MembershipFeeType`. **Empty cell → default fee type** (no warning).
|
||||
**Matched name → that fee type.** **Unmatched name → default fee type + warning** naming the value.
|
||||
|
||||
These columns are resolved against the DB read-only in `prepare` (`ColumnResolver`) for the
|
||||
preview; the actual writes happen in `process_chunk`.
|
||||
|
||||
### Fields not importable (explicitly ignored)
|
||||
|
||||
- **membership_fee_status** — computed from fee cycles; not stored. Fee-status header variants
|
||||
(`Membership Fee Status`, `Bezahlstatus`, `Mitgliedsbeitragsstatus`) and the DE export label
|
||||
`Startdatum Mitgliedsbeitrag` are placed in the `ignored` list and never mapped. (The UI notice
|
||||
names `Groups`/`Gruppen`, `Fee Type`/`Beitragsart`, and the always-ignored `Bezahlstatus`.)
|
||||
|
||||
## Custom Fields
|
||||
|
||||
- Custom field columns are matched by the custom field **name** (not slug), using the same
|
||||
normalization. Member fields take priority on a name collision.
|
||||
- **Custom fields must exist in Mila before import.** Unknown custom-field columns are ignored
|
||||
with a warning; the import still runs.
|
||||
- Empty custom-field cells create no value. Values are trimmed; type-validated per the custom
|
||||
field's `value_type`:
|
||||
- **string** — any text (trimmed).
|
||||
- **integer** — must parse fully (`Integer.parse` with no remainder); e.g. `42`, `-10`.
|
||||
- **boolean** — case-insensitive `true/false`, `1/0`, `yes/no`, `ja/nein`.
|
||||
- **date** — ISO-8601 `YYYY-MM-DD`.
|
||||
- **email** — validated with `EctoCommons.EmailValidator` (same checks as member email).
|
||||
- A value failing type validation fails the row. Error message format:
|
||||
`custom_field: <name> – expected <type>, got: <value>` (type label is the human-readable
|
||||
`FieldTypes.label/1`, with format hints for boolean/date).
|
||||
|
||||
## Validation & Member Creation (`process_chunk/4` → `process_row`)
|
||||
|
||||
Per row: validate → create member → create custom-field values → assign groups. Sequential.
|
||||
|
||||
- **Email** is required and format-validated (`EctoCommons.EmailValidator`, `Mv.Constants.email_validator_checks()`) on a trimmed value. All string member values are trimmed.
|
||||
- **Date fields** (`join_date`, `exit_date`, `membership_fee_start_date`): empty/blank strings are converted to `nil` so Ash accepts them.
|
||||
- Member created via `Mv.Membership.create_member/2`. Custom field values are passed as
|
||||
`custom_field_values` (Ash union `_union_type`/`_union_value` format), omitted when none.
|
||||
- **Errors** are `%MemberCSV.Error{csv_line_number, field, message}`:
|
||||
- `csv_line_number` is the physical line (1-based); never recomputed in this layer.
|
||||
- Validation errors get `field: :email`; Ash errors prefer the field-level error.
|
||||
- **Duplicate email** (unique constraint) is surfaced as a friendly
|
||||
`"email <addr> has already been taken"` message.
|
||||
- **Error capping** (`max_errors`, default 50, tracked across chunks via `existing_error_count`):
|
||||
once the cap is hit, no further errors are collected but **all rows are still processed** and
|
||||
the `failed` count stays accurate; `errors_truncated?` is set and the UI shows a truncation notice.
|
||||
|
||||
## Templates (`ImportTemplateController`)
|
||||
|
||||
- Generated on the fly (not static files), gated by `can?(:create, Member)`.
|
||||
- One header row: standard member columns (localized EN/DE) + `Groups`/`Gruppen` +
|
||||
`Fee Type`/`Beitragsart` + **every existing custom field name** appended, then one example row.
|
||||
- Semicolon-delimited, RFC-4180 quoting; fields run through `MembersCSV.safe_cell/1` to
|
||||
neutralize spreadsheet formula injection (e.g. a custom-field name like `=HYPERLINK(...)`).
|
||||
|
|
|
|||
|
|
@ -1,67 +1,23 @@
|
|||
# Membership Fees - Technical Architecture
|
||||
|
||||
**Project:** Mila - Membership Management System
|
||||
**Feature:** Membership Fee Management
|
||||
**Version:** 1.0
|
||||
**Last Updated:** 2026-01-13
|
||||
**Status:** ✅ Implemented
|
||||
**Feature:** Membership Fee Management — **Status:** Implemented
|
||||
|
||||
Architectural decisions, patterns, module structure, and integration points (no concrete implementation details).
|
||||
|
||||
**Related:** [membership-fee-overview.md](./membership-fee-overview.md) (business logic, worked examples, UI mockups), [database-schema-readme.md](./database-schema-readme.md), [database_schema.dbml](./database_schema.dbml).
|
||||
|
||||
---
|
||||
|
||||
## Purpose
|
||||
## Core Design Decisions
|
||||
|
||||
This document defines the technical architecture for the Membership Fees system. It focuses on architectural decisions, patterns, module structure, and integration points **without** concrete implementation details.
|
||||
|
||||
**Related Documents:**
|
||||
|
||||
- [membership-fee-overview.md](./membership-fee-overview.md) - Business logic and requirements
|
||||
- [database-schema-readme.md](./database-schema-readme.md) - Database documentation
|
||||
- [database_schema.dbml](./database_schema.dbml) - Database schema definition
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Architecture Principles](#architecture-principles)
|
||||
2. [Domain Structure](#domain-structure)
|
||||
3. [Data Architecture](#data-architecture)
|
||||
4. [Business Logic Architecture](#business-logic-architecture)
|
||||
5. [Integration Points](#integration-points)
|
||||
6. [Acceptance Criteria](#acceptance-criteria)
|
||||
7. [Testing Strategy](#testing-strategy)
|
||||
8. [Security Considerations](#security-considerations)
|
||||
9. [Performance Considerations](#performance-considerations)
|
||||
|
||||
---
|
||||
|
||||
## Architecture Principles
|
||||
|
||||
### Core Design Decisions
|
||||
|
||||
1. **Single Responsibility:**
|
||||
- Each module has one clear responsibility
|
||||
- Cycle generation separated from status management
|
||||
- Calendar logic isolated in dedicated module
|
||||
|
||||
2. **No Redundancy:**
|
||||
- No `cycle_end` field (calculated from `cycle_start` + `interval`)
|
||||
- No `interval_type` field (read from `membership_fee_type.interval`)
|
||||
- Eliminates data inconsistencies
|
||||
|
||||
3. **Immutability Where Important:**
|
||||
- `membership_fee_type.interval` cannot be changed after creation
|
||||
- Prevents complex migration scenarios
|
||||
- Enforced via Ash change validation
|
||||
|
||||
4. **Historical Accuracy:**
|
||||
- `amount` stored per cycle for audit trail
|
||||
- Enables tracking of membership fee changes over time
|
||||
- Old cycles retain original amounts
|
||||
|
||||
5. **Calendar-Based Cycles:**
|
||||
- All cycles aligned to calendar boundaries
|
||||
- Simplifies date calculations
|
||||
- Predictable cycle generation
|
||||
1. **No redundant fields:**
|
||||
- No `cycle_end` field — calculated from `cycle_start` + `interval`.
|
||||
- No `interval_type` field — read from `membership_fee_type.interval`.
|
||||
- Eliminates data inconsistencies.
|
||||
2. **Interval immutability:** `membership_fee_type.interval` cannot be changed after creation (enforced via an Ash validation in `Mv.MembershipFees.MembershipFeeType`, and the attribute is omitted from the update action's `accept` list). Prevents complex migration scenarios.
|
||||
3. **Historical accuracy:** `amount` stored per cycle for audit trail — old cycles retain their original amount, so membership-fee changes over time stay traceable.
|
||||
4. **Calendar-based cycles:** all cycles aligned to calendar boundaries; simplifies date math and makes generation predictable.
|
||||
5. **Single responsibility:** cycle generation, status management, and calendar logic live in separate modules.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -69,25 +25,20 @@ This document defines the technical architecture for the Membership Fees system.
|
|||
|
||||
### Ash Domain: `Mv.MembershipFees`
|
||||
|
||||
**Purpose:** Encapsulates all membership fee-related resources and logic
|
||||
Encapsulates all membership-fee resources and logic.
|
||||
|
||||
**Resources:**
|
||||
|
||||
- `MembershipFeeType` - Membership fee type definitions (admin-managed)
|
||||
- `MembershipFeeCycle` - Individual membership fee cycles per member
|
||||
- `MembershipFeeType` — membership fee type definitions (admin-managed).
|
||||
- `MembershipFeeCycle` — individual membership fee cycles per member.
|
||||
|
||||
**Public API:**
|
||||
The domain exposes code interface functions:
|
||||
- `create_membership_fee_type/1`, `list_membership_fee_types/0`, `update_membership_fee_type/2`, `destroy_membership_fee_type/1`
|
||||
- `create_membership_fee_cycle/1`, `list_membership_fee_cycles/0`, `update_membership_fee_cycle/2`, `destroy_membership_fee_cycle/1`
|
||||
**Public API** (code interface): `create/list/update/destroy_membership_fee_type`, `create/list/update/destroy_membership_fee_cycle`.
|
||||
|
||||
**Note:** In LiveViews, direct `Ash.read`, `Ash.create`, `Ash.update`, `Ash.destroy` calls are used with `domain: Mv.MembershipFees` instead of code interface functions. This is acceptable for LiveView forms that use `AshPhoenix.Form`.
|
||||
**Note:** LiveViews use direct `Ash.read/create/update/destroy` with `domain: Mv.MembershipFees` instead of the code interface — acceptable for LiveView forms using `AshPhoenix.Form`.
|
||||
|
||||
**Extensions:**
|
||||
The Member resource is extended with membership fee fields.
|
||||
|
||||
- Member resource extended with membership fee fields
|
||||
|
||||
### Module Organization
|
||||
### Module Map
|
||||
|
||||
```
|
||||
lib/
|
||||
|
|
@ -96,12 +47,12 @@ lib/
|
|||
│ ├── membership_fee_type.ex # MembershipFeeType resource
|
||||
│ ├── membership_fee_cycle.ex # MembershipFeeCycle resource
|
||||
│ └── changes/
|
||||
│ ├── prevent_interval_change.ex # Validates interval immutability
|
||||
│ ├── set_membership_fee_start_date.ex # Auto-sets start date
|
||||
│ └── validate_same_interval.ex # Validates interval match on type change
|
||||
├── mv/
|
||||
│ └── membership_fees/
|
||||
│ ├── cycle_generator.ex # Cycle generation algorithm
|
||||
│ ├── cycle_generation_job.ex # Scheduled cycle generation job
|
||||
│ └── calendar_cycles.ex # Calendar cycle calculations
|
||||
└── membership/
|
||||
└── member.ex # Extended with membership fee relationships
|
||||
|
|
@ -109,623 +60,146 @@ lib/
|
|||
|
||||
### Separation of Concerns
|
||||
|
||||
**Domain Layer (Ash Resources):**
|
||||
|
||||
- Data validation
|
||||
- Relationship management
|
||||
- Policy enforcement
|
||||
- Action definitions
|
||||
|
||||
**Business Logic Layer (`Mv.MembershipFees`):**
|
||||
|
||||
- Cycle generation algorithm
|
||||
- Calendar calculations
|
||||
- Date boundary handling
|
||||
- Status transitions
|
||||
|
||||
**UI Layer (LiveView):**
|
||||
|
||||
- User interaction
|
||||
- Display logic
|
||||
- Authorization checks
|
||||
- Form handling
|
||||
- **Domain layer (Ash resources):** data validation, relationships, policy enforcement, action definitions.
|
||||
- **Business logic (`Mv.MembershipFees`):** cycle generation, calendar calculations, date boundaries, status transitions.
|
||||
- **UI layer (LiveView):** interaction, display, authorization checks, form handling.
|
||||
|
||||
---
|
||||
|
||||
## Data Architecture
|
||||
|
||||
### Database Schema Extensions
|
||||
|
||||
**See:** [database-schema-readme.md](./database-schema-readme.md) and [database_schema.dbml](./database_schema.dbml) for complete schema documentation.
|
||||
See [database-schema-readme.md](./database-schema-readme.md) and [database_schema.dbml](./database_schema.dbml) for complete schema.
|
||||
|
||||
### New Tables
|
||||
|
||||
1. **`membership_fee_types`**
|
||||
- Purpose: Define membership fee types with fixed intervals
|
||||
- Key Constraint: `interval` field immutable after creation
|
||||
- Relationships: has_many members, has_many membership_fee_cycles
|
||||
|
||||
2. **`membership_fee_cycles`**
|
||||
- Purpose: Individual membership fee cycles for members
|
||||
- Key Design: NO `cycle_end` or `interval_type` fields (calculated)
|
||||
- Relationships: belongs_to member, belongs_to membership_fee_type
|
||||
- Composite uniqueness: One cycle per member per cycle_start
|
||||
1. **`membership_fee_types`** — fee types with fixed `interval` (immutable after creation). has_many members, has_many membership_fee_cycles.
|
||||
2. **`membership_fee_cycles`** — per-member cycles. NO `cycle_end`/`interval_type` (calculated). belongs_to member, belongs_to membership_fee_type. Composite uniqueness: one cycle per member per `cycle_start`.
|
||||
|
||||
### Member Table Extensions
|
||||
|
||||
**Fields Added:**
|
||||
|
||||
- `membership_fee_type_id` (FK, NOT NULL with default from settings)
|
||||
- `membership_fee_type_id` (FK, nullable — default applied from settings at the app level)
|
||||
- `membership_fee_start_date` (Date, nullable)
|
||||
|
||||
**Existing Fields Used:**
|
||||
|
||||
- `join_date` - For calculating membership fee start
|
||||
- `exit_date` - For limiting cycle generation
|
||||
- These fields must remain member fields and should not be replaced by custom fields in the future
|
||||
**Existing fields used:** `join_date` (computes membership fee start), `exit_date` (limits cycle generation). These must remain Member fields and should **not** be replaced by custom fields in the future.
|
||||
|
||||
### Settings Integration
|
||||
|
||||
**Global Settings:**
|
||||
|
||||
- `membership_fees.include_joining_cycle` (Boolean)
|
||||
- `membership_fees.default_membership_fee_type_id` (UUID)
|
||||
|
||||
**Storage:** Existing settings mechanism (TBD: dedicated table or configuration resource)
|
||||
Global settings: `membership_fees.include_joining_cycle` (Boolean), `membership_fees.default_membership_fee_type_id` (UUID). Read during cycle generation and member creation; written only via admin UI. Validation: default fee type must exist.
|
||||
|
||||
### Foreign Key Behaviors
|
||||
|
||||
| Relationship | On Delete | Rationale |
|
||||
|--------------|-----------|-----------|
|
||||
| `membership_fee_cycles.member_id → members.id` | CASCADE | Remove membership fee cycles when member deleted |
|
||||
| `membership_fee_cycles.membership_fee_type_id → membership_fee_types.id` | RESTRICT | Prevent membership fee type deletion if cycles exist |
|
||||
| `members.membership_fee_type_id → membership_fee_types.id` | RESTRICT | Prevent membership fee type deletion if assigned to members |
|
||||
| `membership_fee_cycles.member_id → members.id` | CASCADE | Remove cycles when member deleted |
|
||||
| `membership_fee_cycles.membership_fee_type_id → membership_fee_types.id` | RESTRICT | Prevent fee type deletion if cycles exist |
|
||||
| `members.membership_fee_type_id → membership_fee_types.id` | RESTRICT | Prevent fee type deletion if assigned to members |
|
||||
|
||||
---
|
||||
|
||||
## Business Logic Architecture
|
||||
|
||||
### Cycle Generation System
|
||||
### Cycle Generation — `Mv.MembershipFees.CycleGenerator`
|
||||
|
||||
**Component:** `Mv.MembershipFees.CycleGenerator`
|
||||
Calculates which cycles should exist for a member, generates the missing ones (idempotent — skips existing), respects `membership_fee_start_date` and `exit_date` boundaries, and uses **PostgreSQL advisory locks per member** to prevent race conditions.
|
||||
|
||||
**Responsibilities:**
|
||||
**Triggers:** fee type assigned (Ash change); member created with fee type (Ash change); scheduled job (daily/weekly cron); admin manual regeneration (UI).
|
||||
|
||||
- Calculate which cycles should exist for a member
|
||||
- Generate missing cycles
|
||||
- Respect membership_fee_start_date and exit_date boundaries
|
||||
- Skip existing cycles (idempotent)
|
||||
- Use PostgreSQL advisory locks per member to prevent race conditions
|
||||
**Algorithm:**
|
||||
|
||||
**Triggers:**
|
||||
|
||||
1. Member membership fee type assigned (via Ash change)
|
||||
2. Member created with membership fee type (via Ash change)
|
||||
3. Scheduled job runs (daily/weekly cron)
|
||||
4. Admin manual regeneration (UI action)
|
||||
|
||||
**Algorithm Steps:**
|
||||
|
||||
1. Retrieve member with membership fee type and dates
|
||||
1. Retrieve member with fee type and dates.
|
||||
2. Determine generation start point:
|
||||
- If NO cycles exist: Start from `membership_fee_start_date` (or calculated from `join_date`)
|
||||
- If cycles exist: Start from the cycle AFTER the last existing one
|
||||
3. Generate all cycle starts from the determined start point to today (or `exit_date`)
|
||||
4. Create new cycles with current membership fee type's amount
|
||||
5. Use PostgreSQL advisory locks per member to prevent race conditions
|
||||
- No cycles exist → start from `membership_fee_start_date` (or calculated from `join_date`).
|
||||
- Cycles exist → start from the cycle AFTER the last existing one.
|
||||
3. Generate all cycle starts from that point to today (or `exit_date`).
|
||||
4. Create new cycles with the current fee type's amount.
|
||||
|
||||
**Edge Case Handling:**
|
||||
**Edge cases:**
|
||||
|
||||
- If membership_fee_start_date is NULL: Calculate from join_date + global setting
|
||||
- If exit_date is set: Stop generation at exit_date
|
||||
- If membership fee type changes: Handled separately by regeneration logic
|
||||
- **Gap Handling:** If cycles were explicitly deleted (gaps exist), they are NOT recreated.
|
||||
The generator always continues from the cycle AFTER the last existing cycle, regardless of gaps.
|
||||
- `membership_fee_start_date` NULL → calculate from `join_date` + global setting.
|
||||
- `exit_date` set → stop generation at `exit_date`.
|
||||
- Fee type changes → handled separately by regeneration logic.
|
||||
- **Gap handling:** if cycles were explicitly deleted (gaps exist), they are **NOT** recreated. The generator always continues from the cycle AFTER the last existing cycle, regardless of gaps.
|
||||
|
||||
### Calendar Cycle Calculations
|
||||
### Calendar Cycles — `Mv.MembershipFees.CalendarCycles`
|
||||
|
||||
**Component:** `Mv.MembershipFees.CalendarCycles`
|
||||
Calculates cycle boundaries by interval, the current cycle, the last completed cycle, and `cycle_end` from `cycle_start` + interval.
|
||||
|
||||
**Responsibilities:**
|
||||
**Functions (high-level):** `calculate_cycle_start/2,3`, `calculate_cycle_end/2`, `next_cycle_start/2`, `current_cycle?/2,3`, `last_completed_cycle?/2,3`.
|
||||
|
||||
- Calculate cycle boundaries based on interval type
|
||||
- Determine current cycle
|
||||
- Determine last completed cycle
|
||||
- Calculate cycle_end from cycle_start + interval
|
||||
**Interval logic:**
|
||||
|
||||
**Functions (high-level):**
|
||||
- **Monthly:** 1st of month → last day of month.
|
||||
- **Quarterly:** 1st of quarter (Jan/Apr/Jul/Oct) → last day of quarter.
|
||||
- **Half-yearly:** 1st of half (Jan/Jul) → last day of half.
|
||||
- **Yearly:** Jan 1 → Dec 31.
|
||||
|
||||
- `calculate_cycle_start/3` - Given date and interval, find cycle start
|
||||
- `calculate_cycle_end/2` - Given cycle_start and interval, calculate end
|
||||
- `next_cycle_start/2` - Given cycle_start and interval, find next
|
||||
- `is_current_cycle?/2` - Check if cycle contains today
|
||||
- `is_last_completed_cycle?/2` - Check if cycle just ended
|
||||
### Status Management — Ash actions on `MembershipFeeCycle`
|
||||
|
||||
**Interval Logic:**
|
||||
Simple state machine unpaid ↔ paid ↔ suspended; all transitions allowed; permissions checked via Ash policies. Actions: `mark_as_paid`, `mark_as_suspended`, `mark_as_unpaid` (error correction). `bulk_mark_as_paid` is low priority / future.
|
||||
|
||||
- **Monthly:** Start = 1st of month, End = last day of month
|
||||
- **Quarterly:** Start = 1st of quarter (Jan/Apr/Jul/Oct), End = last day of quarter
|
||||
- **Half-yearly:** Start = 1st of half (Jan/Jul), End = last day of half
|
||||
- **Yearly:** Start = Jan 1st, End = Dec 31st
|
||||
### Membership Fee Type Change — Ash change on `Member.membership_fee_type_id`
|
||||
|
||||
### Status Management
|
||||
**Validation:** new type must have the same interval as the old type; different interval is rejected (MVP constraint).
|
||||
|
||||
**Component:** Ash actions on `MembershipFeeCycle`
|
||||
**Side effects on allowed change:** keep all existing cycles; find future unpaid cycles, delete them, regenerate with the new `membership_fee_type_id` and amount.
|
||||
|
||||
**Status Transitions:**
|
||||
**Implementation pattern:**
|
||||
|
||||
- Simple state machine: unpaid ↔ paid ↔ suspended
|
||||
- No complex validation (all transitions allowed)
|
||||
- Permissions checked via Ash policies
|
||||
- Ash change module validates; `after_action` hook triggers regeneration synchronously.
|
||||
- **Regeneration runs in the same transaction as the member update** to ensure atomicity. CycleGenerator uses advisory locks and transactions internally to prevent races.
|
||||
|
||||
**Actions Required:**
|
||||
**Validation behavior:**
|
||||
|
||||
- `mark_as_paid` - Set status to :paid
|
||||
- `mark_as_suspended` - Set status to :suspended
|
||||
- `mark_as_unpaid` - Set status to :unpaid (error correction)
|
||||
|
||||
**Bulk Operations:**
|
||||
|
||||
- `bulk_mark_as_paid` - Mark multiple cycles as paid (efficiency)
|
||||
- low priority, can be a future issue
|
||||
|
||||
### Membership Fee Type Change Handling
|
||||
|
||||
**Component:** Ash change on `Member.membership_fee_type_id`
|
||||
|
||||
**Validation:**
|
||||
|
||||
- Check if new type has same interval as old type
|
||||
- If different: Reject change (MVP constraint)
|
||||
- If same: Allow change
|
||||
|
||||
**Side Effects on Allowed Change:**
|
||||
|
||||
1. Keep all existing cycles unchanged
|
||||
2. Find future unpaid cycles
|
||||
3. Delete future unpaid cycles
|
||||
4. Regenerate cycles with new membership_fee_type_id and amount
|
||||
|
||||
**Implementation Pattern:**
|
||||
|
||||
- Use Ash change module to validate
|
||||
- Use after_action hook to trigger regeneration synchronously
|
||||
- Regeneration runs in the same transaction as the member update to ensure atomicity
|
||||
- CycleGenerator uses advisory locks and transactions internally to prevent race conditions
|
||||
|
||||
**Validation Behavior:**
|
||||
|
||||
- Fail-closed: If membership fee types cannot be loaded during validation, the change is rejected with a validation error
|
||||
- Nil assignment prevention: Attempts to set membership_fee_type_id to nil are rejected when a current type exists
|
||||
- **Fail-closed:** if fee types cannot be loaded during validation, the change is rejected with a validation error.
|
||||
- **Nil prevention:** setting `membership_fee_type_id` to nil is rejected when a current type exists.
|
||||
|
||||
---
|
||||
|
||||
## Integration Points
|
||||
|
||||
### Member Resource Integration
|
||||
### Member Resource
|
||||
|
||||
**Extension Points:**
|
||||
Extension points: fields via migration; relationships (belongs_to, has_many); calculations (current_cycle_status, overdue_count); changes (auto-set `membership_fee_start_date`, validate interval). Backward compatible — new fields nullable/defaulted; existing members get the default fee type from settings.
|
||||
|
||||
1. Add fields via migration
|
||||
2. Add relationships (belongs_to, has_many)
|
||||
3. Add calculations (current_cycle_status, overdue_count)
|
||||
4. Add changes (auto-set membership_fee_start_date, validate interval)
|
||||
### Settings System
|
||||
|
||||
**Backward Compatibility:**
|
||||
Store two global settings, admin UI to modify, default values if unset, validation (default fee type must exist).
|
||||
|
||||
- New fields nullable or with defaults
|
||||
- Existing members get default membership fee type from settings
|
||||
- No breaking changes to existing member functionality
|
||||
### Permission System — Implemented
|
||||
|
||||
### Settings System Integration
|
||||
See [roles-and-permissions-architecture.md](./roles-and-permissions-architecture.md) for the full matrix and policy patterns.
|
||||
|
||||
**Requirements:**
|
||||
**PermissionSets (`lib/mv/authorization/permission_sets.ex`):**
|
||||
|
||||
- Store two global settings
|
||||
- Provide UI for admin to modify
|
||||
- Default values if not set
|
||||
- Validation (e.g., default membership fee type must exist)
|
||||
- **MembershipFeeType:** all sets read (:all); only admin has create/update/destroy (:all).
|
||||
- **MembershipFeeCycle:** all read (:all); read_only has read only; normal_user and admin have read + create + update + destroy (:all).
|
||||
- **Manual "Regenerate Cycles" (UI + server):** the "Regenerate Cycles" button in the member detail view is shown to users with MembershipFeeCycle create permission (normal_user and admin). UI access is gated by `can_create_cycle`. The LiveView handler **also enforces `can?(:create, MembershipFeeCycle)` server-side** before running regeneration (so e.g. a read_only user cannot trigger it via DevTools). Regeneration runs with the system actor.
|
||||
|
||||
**Access Pattern:**
|
||||
**Resource policies:**
|
||||
|
||||
- Read settings during cycle generation
|
||||
- Read settings during member creation
|
||||
- Write settings only via admin UI
|
||||
|
||||
### Permission System Integration
|
||||
|
||||
**Status:** ✅ Implemented. See [roles-and-permissions-architecture.md](./roles-and-permissions-architecture.md) for the full permission matrix and policy patterns.
|
||||
|
||||
**PermissionSets (lib/mv/authorization/permission_sets.ex):**
|
||||
|
||||
- **MembershipFeeType:** All permission sets can read (:all); only admin has create/update/destroy (:all).
|
||||
- **MembershipFeeCycle:** All can read (:all); read_only has read only; normal_user and admin have read + create + update + destroy (:all).
|
||||
- **Manual "Regenerate Cycles" (UI + server):** The "Regenerate Cycles" button in the member detail view is shown to users who have MembershipFeeCycle create permission (normal_user and admin). UI access is gated by `can_create_cycle`. The LiveView handler also enforces `can?(:create, MembershipFeeCycle)` server-side before running regeneration (so e.g. a read_only user cannot trigger it via DevTools). Regeneration runs with system actor.
|
||||
|
||||
**Resource Policies:**
|
||||
|
||||
- **MembershipFeeType** (`lib/membership_fees/membership_fee_type.ex`): `authorizers: [Ash.Policy.Authorizer]`, single policy with `HasPermission` for read/create/update/destroy.
|
||||
- **MembershipFeeCycle** (`lib/membership_fees/membership_fee_cycle.ex`): Same pattern; update includes mark_as_paid, mark_as_suspended, mark_as_unpaid.
|
||||
- `MembershipFeeType` (`lib/membership_fees/membership_fee_type.ex`): `authorizers: [Ash.Policy.Authorizer]`, single policy with `HasPermission` for read/create/update/destroy.
|
||||
- `MembershipFeeCycle` (`lib/membership_fees/membership_fee_cycle.ex`): same pattern; update includes mark_as_paid, mark_as_suspended, mark_as_unpaid.
|
||||
|
||||
### LiveView Integration
|
||||
|
||||
**New LiveViews Required:**
|
||||
**New:** MembershipFeeType index/form (admin); MembershipFeeCycle table component in the member detail view — implemented as `MvWeb.MemberLive.Show.MembershipFeesComponent` (displays all cycles with status management, amount editing, and manual regeneration for normal_user and admin); Settings form section (admin); member-list status column.
|
||||
|
||||
1. MembershipFeeType index/form (admin)
|
||||
2. MembershipFeeCycle table component (member detail view)
|
||||
- Implemented as `MvWeb.MemberLive.Show.MembershipFeesComponent`
|
||||
- Displays all cycles in a table with status management
|
||||
- Allows changing cycle status, editing amounts, and manually regenerating cycles (normal_user and admin)
|
||||
3. Settings form section (admin)
|
||||
4. Member list column (membership fee status)
|
||||
|
||||
**Existing LiveViews to Extend:**
|
||||
|
||||
- Member detail view: Add membership fees section
|
||||
- Member list view: Add status column
|
||||
- Settings page: Add membership fees section
|
||||
|
||||
**Authorization Helpers:**
|
||||
|
||||
- Use existing `can?/3` helper for UI conditionals
|
||||
- Check permissions before showing actions
|
||||
**Extended:** member detail view (membership fees section), member list view (status column), settings page (membership fees section). Use the existing `can?/3` helper for UI conditionals.
|
||||
|
||||
---
|
||||
|
||||
## Acceptance Criteria
|
||||
## Performance Notes
|
||||
|
||||
### MembershipFeeType Resource
|
||||
**Indexes:** `membership_fee_cycles` on `member_id`, `membership_fee_type_id`, `status`, `cycle_start`, composite unique `(member_id, cycle_start)`; `members(membership_fee_type_id)`.
|
||||
|
||||
**AC-MFT-1:** Admin can create membership fee type with name, amount, interval, description
|
||||
**AC-MFT-2:** Interval field is immutable after creation (validation error on change attempt)
|
||||
**AC-MFT-3:** Admin can update name, amount, description (but not interval)
|
||||
**AC-MFT-4:** Cannot delete membership fee type if assigned to members
|
||||
**AC-MFT-5:** Cannot delete membership fee type if cycles exist referencing it
|
||||
**AC-MFT-6:** Interval must be one of: monthly, quarterly, half_yearly, yearly
|
||||
**Query:** preload fee type with cycles to avoid N+1; `cycle_end` and `current_cycle_status` are Ash calculations (lazy, not stored); paginate cycle lists > 50.
|
||||
|
||||
### MembershipFeeCycle Resource
|
||||
|
||||
**AC-MFC-1:** Cycle has cycle_start, status, amount, notes, member_id, membership_fee_type_id
|
||||
**AC-MFC-2:** cycle_end is calculated, not stored
|
||||
**AC-MFC-3:** Status defaults to :unpaid
|
||||
**AC-MFC-4:** One cycle per member per cycle_start (uniqueness constraint)
|
||||
**AC-MFC-5:** Amount is set at generation time from membership_fee_type.amount
|
||||
**AC-MFC-6:** Cycles cascade delete when member deleted
|
||||
**AC-MFC-7:** Admin/Treasurer can change status
|
||||
**AC-MFC-8:** Member can read own cycles
|
||||
|
||||
### Member Extensions
|
||||
|
||||
**AC-M-1:** Member has membership_fee_type_id field (NOT NULL with default)
|
||||
**AC-M-2:** Member has membership_fee_start_date field (nullable)
|
||||
**AC-M-3:** New members get default membership fee type from global setting
|
||||
**AC-M-4:** membership_fee_start_date auto-set based on join_date and global setting
|
||||
**AC-M-5:** Admin can manually override membership_fee_start_date
|
||||
**AC-M-6:** Cannot change to membership fee type with different interval (MVP)
|
||||
|
||||
### Cycle Generation
|
||||
|
||||
**AC-CG-1:** Cycles generated when member gets membership fee type
|
||||
**AC-CG-2:** Cycles generated when member created (via change hook)
|
||||
**AC-CG-3:** Scheduled job generates missing cycles daily
|
||||
**AC-CG-4:** Generation respects membership_fee_start_date
|
||||
**AC-CG-5:** Generation stops at exit_date if member exited
|
||||
**AC-CG-6:** Generation is idempotent (skips existing cycles)
|
||||
**AC-CG-7:** Cycles align to calendar boundaries (1st of month/quarter/half/year)
|
||||
**AC-CG-8:** Amount comes from membership_fee_type at generation time
|
||||
|
||||
### Calendar Logic
|
||||
|
||||
**AC-CL-1:** Monthly cycles: 1st to last day of month
|
||||
**AC-CL-2:** Quarterly cycles: 1st of Jan/Apr/Jul/Oct to last day of quarter
|
||||
**AC-CL-3:** Half-yearly cycles: 1st of Jan/Jul to last day of half
|
||||
**AC-CL-4:** Yearly cycles: Jan 1 to Dec 31
|
||||
**AC-CL-5:** cycle_end calculated correctly for all interval types
|
||||
**AC-CL-6:** Current cycle determined correctly based on today's date
|
||||
**AC-CL-7:** Last completed cycle determined correctly
|
||||
|
||||
### Membership Fee Type Change
|
||||
|
||||
**AC-TC-1:** Can change to type with same interval
|
||||
**AC-TC-2:** Cannot change to type with different interval (error message)
|
||||
**AC-TC-3:** On allowed change: future unpaid cycles regenerated
|
||||
**AC-TC-4:** On allowed change: paid/suspended cycles unchanged
|
||||
**AC-TC-5:** On allowed change: amount updated to new type's amount
|
||||
**AC-TC-6:** Change is atomic (transaction) - ✅ Implemented: Regeneration runs synchronously in the same transaction as the member update
|
||||
|
||||
### Settings
|
||||
|
||||
**AC-S-1:** Global setting: include_joining_cycle (boolean, default true)
|
||||
**AC-S-2:** Global setting: default_membership_fee_type_id (UUID, required)
|
||||
**AC-S-3:** Admin can modify settings via UI
|
||||
**AC-S-4:** Settings validated (e.g., default membership fee type must exist)
|
||||
**AC-S-5:** Settings applied to new members immediately
|
||||
|
||||
### UI - Member List
|
||||
|
||||
**AC-UI-ML-1:** New column shows membership fee status
|
||||
**AC-UI-ML-2:** Default: Shows last completed cycle status
|
||||
**AC-UI-ML-3:** Optional: Toggle to show current cycle status
|
||||
**AC-UI-ML-4:** Color coding: green (paid), red (unpaid), gray (suspended)
|
||||
**AC-UI-ML-5:** Filter: Unpaid in last cycle
|
||||
**AC-UI-ML-6:** Filter: Unpaid in current cycle
|
||||
|
||||
### UI - Member Detail
|
||||
|
||||
**AC-UI-MD-1:** Membership fees section shows all cycles
|
||||
**AC-UI-MD-2:** Table columns: Cycle, Interval, Amount, Status, Actions
|
||||
**AC-UI-MD-3:** Checkbox per cycle for bulk marking (low prio)
|
||||
**AC-UI-MD-4:** "Mark selected as paid" button
|
||||
**AC-UI-MD-5:** Dropdown to change membership fee type (same interval only)
|
||||
**AC-UI-MD-6:** Warning if different interval selected
|
||||
**AC-UI-MD-7:** Only show actions if user has permission
|
||||
|
||||
### UI - Membership Fee Types Admin
|
||||
|
||||
**AC-UI-CTA-1:** List all membership fee types
|
||||
**AC-UI-CTA-2:** Show: Name, Amount, Interval, Member count
|
||||
**AC-UI-CTA-3:** Create new membership fee type form
|
||||
**AC-UI-CTA-4:** Edit form: Name, Amount, Description editable
|
||||
**AC-UI-CTA-5:** Edit form: Interval grayed out (not editable)
|
||||
**AC-UI-CTA-6:** Warning on amount change (explain impact)
|
||||
**AC-UI-CTA-7:** Cannot delete if members assigned
|
||||
**AC-UI-CTA-8:** Only admin can access
|
||||
|
||||
### UI - Settings Admin
|
||||
|
||||
**AC-UI-SA-1:** Membership fees section in settings
|
||||
**AC-UI-SA-2:** Dropdown to select default membership fee type
|
||||
**AC-UI-SA-3:** Checkbox: Include joining cycle
|
||||
**AC-UI-SA-4:** Explanatory text with examples
|
||||
**AC-UI-SA-5:** Save button with validation
|
||||
|
||||
---
|
||||
|
||||
## Testing Strategy
|
||||
|
||||
### Unit Testing
|
||||
|
||||
**Cycle Generator Tests:**
|
||||
|
||||
- Correct cycle_start calculation for all interval types
|
||||
- Correct cycle count from start to end date
|
||||
- Respects membership_fee_start_date boundary
|
||||
- Respects exit_date boundary
|
||||
- Skips existing cycles (idempotent)
|
||||
- Does not fill gaps when cycles were deleted
|
||||
- Handles edge dates (year boundaries, leap years)
|
||||
|
||||
**Calendar Cycles Tests:**
|
||||
|
||||
- Cycle boundaries correct for all intervals
|
||||
- cycle_end calculation correct
|
||||
- Current cycle detection
|
||||
- Last completed cycle detection
|
||||
- Next cycle calculation
|
||||
|
||||
**Validation Tests:**
|
||||
|
||||
- Interval immutability enforced
|
||||
- Same interval validation on type change
|
||||
- Status transitions allowed
|
||||
- Uniqueness constraints enforced
|
||||
|
||||
### Integration Testing
|
||||
|
||||
**Cycle Generation Flow:**
|
||||
|
||||
- Member creation triggers generation
|
||||
- Type assignment triggers generation
|
||||
- Type change regenerates future cycles
|
||||
- Scheduled job generates missing cycles
|
||||
- Left member stops generation
|
||||
|
||||
**Status Management Flow:**
|
||||
|
||||
- Mark single cycle as paid
|
||||
- Bulk mark multiple cycles (low prio)
|
||||
- Status transitions work
|
||||
- Permissions enforced
|
||||
|
||||
**Membership Fee Type Management:**
|
||||
|
||||
- Create type
|
||||
- Update amount (regeneration triggered)
|
||||
- Cannot update interval
|
||||
- Cannot delete if in use
|
||||
|
||||
### LiveView Testing
|
||||
|
||||
**Member List:**
|
||||
|
||||
- Status column displays correctly
|
||||
- Toggle between last/current works
|
||||
- Filters work correctly
|
||||
- Color coding applied
|
||||
|
||||
**Member Detail:**
|
||||
|
||||
- Cycles table displays all cycles
|
||||
- Checkboxes work
|
||||
- Bulk marking works (low prio)
|
||||
- Membership fee type change validation works
|
||||
- Actions only shown with permission
|
||||
|
||||
**Admin UI:**
|
||||
|
||||
- Type CRUD works
|
||||
- Settings save correctly
|
||||
- Validations display errors
|
||||
- Only authorized users can access
|
||||
|
||||
### Edge Case Testing
|
||||
|
||||
**Interval Change Attempt:**
|
||||
|
||||
- Error message displayed
|
||||
- No data modified
|
||||
- User can cancel/choose different type
|
||||
|
||||
**Exit with Unpaid:**
|
||||
|
||||
- Warning shown
|
||||
- Option to suspend offered
|
||||
- Exit completes correctly
|
||||
|
||||
**Amount Change:**
|
||||
|
||||
- Warning displayed
|
||||
- Only future unpaid regenerated
|
||||
- Historical cycles unchanged
|
||||
|
||||
**Date Boundaries:**
|
||||
|
||||
- Today = cycle start handled
|
||||
- Today = cycle end handled
|
||||
- Leap year handled
|
||||
|
||||
### Performance Testing
|
||||
|
||||
**Cycle Generation:**
|
||||
|
||||
- Generate 10 years of monthly cycles: < 100ms
|
||||
- Generate for 1000 members: < 5 seconds
|
||||
- Idempotent check efficient (no full scan)
|
||||
|
||||
**Member List Query:**
|
||||
|
||||
- With status column: < 200ms for 1000 members
|
||||
- Filters applied efficiently
|
||||
- No N+1 queries
|
||||
|
||||
---
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Authorization
|
||||
|
||||
**Permissions Required:**
|
||||
|
||||
- Membership fee type management: Admin only
|
||||
- Membership fee cycle status changes: Admin + Treasurer
|
||||
- View all cycles: Admin + Treasurer + Board
|
||||
- View own cycles: All authenticated users
|
||||
|
||||
**Policy Enforcement:**
|
||||
|
||||
- All actions protected by Ash policies
|
||||
- UI shows/hides based on permissions
|
||||
- Backend validates permissions (never trust UI alone)
|
||||
|
||||
### Data Integrity
|
||||
|
||||
**Validation Layers:**
|
||||
|
||||
1. Database constraints (NOT NULL, UNIQUE, CHECK)
|
||||
2. Ash validations (business rules)
|
||||
3. UI validations (user experience)
|
||||
|
||||
**Immutability Protection:**
|
||||
|
||||
- Interval change prevented at multiple layers
|
||||
- Cycle amounts immutable (audit trail)
|
||||
- Settings changes logged (future)
|
||||
|
||||
### Audit Trail
|
||||
|
||||
**Tracked Information:**
|
||||
|
||||
- Cycle status changes (who, when) - future enhancement
|
||||
- Membership fee type amount changes (implicit via cycle amounts)
|
||||
|
||||
---
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
### Database Indexes
|
||||
|
||||
**Required Indexes:**
|
||||
|
||||
- `membership_fee_cycles(member_id)` - For member cycle lookups
|
||||
- `membership_fee_cycles(membership_fee_type_id)` - For type queries
|
||||
- `membership_fee_cycles(status)` - For unpaid filters
|
||||
- `membership_fee_cycles(cycle_start)` - For date range queries
|
||||
- `membership_fee_cycles(member_id, cycle_start)` - Composite unique index
|
||||
- `members(membership_fee_type_id)` - For type membership count
|
||||
|
||||
### Query Optimization
|
||||
|
||||
**Preloading:**
|
||||
|
||||
- Load membership_fee_type with cycles (avoid N+1)
|
||||
- Load cycles when displaying member detail
|
||||
- Use Ash's load for efficient preloading
|
||||
|
||||
**Calculated Fields:**
|
||||
|
||||
- cycle_end calculated on-demand (not stored)
|
||||
- current_cycle_status calculated when needed
|
||||
- Use Ash calculations for lazy evaluation
|
||||
|
||||
**Pagination:**
|
||||
|
||||
- Cycle list paginated if > 50 cycles
|
||||
- Member list already paginated
|
||||
|
||||
### Caching Strategy
|
||||
|
||||
**No caching needed in MVP:**
|
||||
|
||||
- Membership fee types rarely change
|
||||
- Cycle queries are fast
|
||||
- Settings read infrequently
|
||||
|
||||
**Future caching if needed:**
|
||||
|
||||
- Cache settings in application memory
|
||||
- Cache membership fee types list
|
||||
- Invalidate on change
|
||||
|
||||
### Scheduled Job Performance
|
||||
|
||||
**Cycle Generation Job:**
|
||||
|
||||
- Run daily or weekly (not hourly)
|
||||
- Batch members (process 100 at a time)
|
||||
- Skip members with no changes
|
||||
- Log failures for retry
|
||||
**No caching in MVP** (fee types rarely change, queries fast). Scheduled generation job: run daily/weekly, batch members, skip unchanged, log failures for retry.
|
||||
|
||||
---
|
||||
|
||||
## Future Enhancements
|
||||
|
||||
### Phase 2: Interval Change Support
|
||||
|
||||
**Architecture Changes:**
|
||||
|
||||
- Add logic to handle cycle overlaps
|
||||
- Calculate prorata amounts if needed
|
||||
- More complex validation
|
||||
- Migration path for existing cycles
|
||||
|
||||
### Phase 3: Payment Details
|
||||
|
||||
**Architecture Changes:**
|
||||
|
||||
- Add PaymentTransaction resource
|
||||
- Link transactions to cycles
|
||||
- Support multiple payments per cycle
|
||||
- Reconciliation logic
|
||||
|
||||
### Phase 4: vereinfacht.digital Integration
|
||||
|
||||
**Architecture Changes:**
|
||||
|
||||
- External API client module
|
||||
- Webhook handling for transactions
|
||||
- Automatic matching logic
|
||||
- Manual review interface
|
||||
|
||||
---
|
||||
|
||||
**End of Architecture Document**
|
||||
- **Phase 2 — Interval change support:** cycle-overlap logic, prorata, more validation, migration path for existing cycles.
|
||||
- **Phase 3 — Payment details:** `PaymentTransaction` resource linked to cycles, multiple payments per cycle, reconciliation.
|
||||
- **Phase 4 — vereinfacht.digital integration:** external API client, webhook handling, automatic matching, manual review.
|
||||
|
|
|
|||
|
|
@ -1,50 +1,20 @@
|
|||
# Membership Fees - Overview
|
||||
|
||||
**Project:** Mila - Membership Management System
|
||||
**Feature:** Membership Fee Management
|
||||
**Version:** 1.0
|
||||
**Last Updated:** 2026-01-13
|
||||
**Status:** ✅ Implemented
|
||||
**Feature:** Membership Fee Management — **Status:** Implemented
|
||||
|
||||
---
|
||||
|
||||
## Purpose
|
||||
|
||||
This document provides a comprehensive overview of the Membership Fees system. It covers business logic, data model, UI/UX design, and technical architecture in a concise, bullet-point format.
|
||||
|
||||
**For detailed implementation:** See [membership-fee-implementation-plan.md](./membership-fee-implementation-plan.md) (created after concept iterations)
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Core Principle](#core-principle)
|
||||
2. [Terminology](#terminology)
|
||||
3. [Data Model](#data-model)
|
||||
4. [Business Logic](#business-logic)
|
||||
5. [UI/UX Design](#uiux-design)
|
||||
6. [Edge Cases](#edge-cases)
|
||||
7. [Technical Integration](#technical-integration)
|
||||
8. [Implementation Scope](#implementation-scope)
|
||||
Coarse, business-oriented entry point for the Membership Fees system: terminology, worked examples, and UI/UX. For architecture (data model, FK behaviors, module map, generation algorithm, policies) see [membership-fee-architecture.md](./membership-fee-architecture.md).
|
||||
|
||||
---
|
||||
|
||||
## Core Principle
|
||||
|
||||
**Maximum Simplicity:**
|
||||
|
||||
- Minimal complexity
|
||||
- Clear data model without redundancies
|
||||
- Intuitive operation
|
||||
- Calendar cycle-based (Month/Quarter/Half-Year/Year)
|
||||
Maximum simplicity: minimal complexity, clear data model without redundancies, intuitive operation, calendar-cycle-based (Month / Quarter / Half-Year / Year).
|
||||
|
||||
---
|
||||
|
||||
## Terminology
|
||||
## Terminology (German ↔ English)
|
||||
|
||||
### German ↔ English
|
||||
|
||||
**Core Entities:**
|
||||
**Core entities:**
|
||||
|
||||
- Beitragsart ↔ Membership Fee Type
|
||||
- Beitragszyklus ↔ Membership Fee Cycle
|
||||
|
|
@ -56,14 +26,14 @@ This document provides a comprehensive overview of the Membership Fees system. I
|
|||
- unbezahlt ↔ unpaid
|
||||
- ausgesetzt ↔ suspended / waived
|
||||
|
||||
**Intervals (Frequenz / Payment Frequency):**
|
||||
**Intervals (Frequenz / payment frequency):**
|
||||
|
||||
- monatlich ↔ monthly
|
||||
- quartalsweise ↔ quarterly
|
||||
- halbjährlich ↔ half-yearly / semi-annually
|
||||
- jährlich ↔ yearly / annually
|
||||
|
||||
**UI Elements:**
|
||||
**UI elements:**
|
||||
|
||||
- "Letzter Zyklus" ↔ "Last Cycle" (e.g., 2023 when in 2024)
|
||||
- "Aktueller Zyklus" ↔ "Current Cycle" (e.g., 2024)
|
||||
|
|
@ -72,112 +42,39 @@ This document provides a comprehensive overview of the Membership Fees system. I
|
|||
|
||||
---
|
||||
|
||||
## Data Model
|
||||
## Data Model (summary)
|
||||
|
||||
### Membership Fee Type (MembershipFeeType)
|
||||
Three entities — full schema, FK on-delete behaviors, and design rationale are in [membership-fee-architecture.md](./membership-fee-architecture.md).
|
||||
|
||||
```
|
||||
- id (UUID)
|
||||
- name (String) - e.g., "Regular", "Reduced", "Student"
|
||||
- amount (Decimal) - Membership fee amount in Euro
|
||||
- interval (Enum) - :monthly, :quarterly, :half_yearly, :yearly
|
||||
- description (Text, optional)
|
||||
```
|
||||
- **MembershipFeeType:** name, amount (€), `interval` (:monthly/:quarterly/:half_yearly/:yearly), optional description. `interval` is **IMMUTABLE** after creation; admin can change only name/amount/description; on amount change, future unpaid cycles regenerate with the new amount.
|
||||
- **MembershipFeeCycle:** member_id, membership_fee_type_id, `cycle_start` (calendar start: 01.01., 01.04., 01.07., 01.10., …), status (:unpaid default / :paid / :suspended), `amount` (captured at generation time → history when type changes), optional notes. NO `cycle_end` (derived from `cycle_start` + interval), NO `interval_type` (read from the fee type).
|
||||
- **Member extensions:** `membership_fee_type_id` (FK, nullable — default applied from settings at the app level), `membership_fee_start_date` (Date, nullable), plus the existing `exit_date`.
|
||||
|
||||
**Important:**
|
||||
**Calendar cycle logic:** Monthly 01.01.–31.01., etc. · Quarterly 01.01.–31.03., 01.04.–30.06., 01.07.–30.09., 01.10.–31.12. · Half-yearly 01.01.–30.06., 01.07.–31.12. · Yearly 01.01.–31.12.
|
||||
|
||||
- `interval` is **IMMUTABLE** after creation!
|
||||
- Admin can only change `name`, `amount`, `description`
|
||||
- On change: Future unpaid cycles regenerated with new amount
|
||||
### `membership_fee_start_date` derivation
|
||||
|
||||
### Membership Fee Cycle (MembershipFeeCycle)
|
||||
Auto-set from global setting `include_joining_cycle`:
|
||||
|
||||
```
|
||||
- id (UUID)
|
||||
- member_id (FK → members.id)
|
||||
- membership_fee_type_id (FK → membership_fee_types.id)
|
||||
- cycle_start (Date) - Calendar cycle start (01.01., 01.04., 01.07., 01.10., etc.)
|
||||
- status (Enum) - :unpaid (default), :paid, :suspended
|
||||
- amount (Decimal) - Membership fee amount at generation time (history when type changes)
|
||||
- notes (Text, optional) - Admin notes
|
||||
```
|
||||
- `include_joining_cycle = true` → first day of the joining month/quarter/year (member pays from the joining cycle).
|
||||
- `include_joining_cycle = false` → first day of the NEXT cycle after joining.
|
||||
|
||||
**Important:**
|
||||
Can be manually overridden by admin. There is intentionally **no** `include_joining_cycle` field on Member — `membership_fee_start_date` makes it unnecessary.
|
||||
|
||||
- **NO** `cycle_end` - calculated from `cycle_start` + `interval`
|
||||
- **NO** `interval_type` - read from `membership_fee_type.interval`
|
||||
- Avoids redundancy and inconsistencies!
|
||||
### Global settings
|
||||
|
||||
**Calendar Cycle Logic:**
|
||||
|
||||
- Monthly: 01.01. - 31.01., 01.02. - 28./29.02., etc.
|
||||
- Quarterly: 01.01. - 31.03., 01.04. - 30.06., 01.07. - 30.09., 01.10. - 31.12.
|
||||
- Half-yearly: 01.01. - 30.06., 01.07. - 31.12.
|
||||
- Yearly: 01.01. - 31.12.
|
||||
|
||||
### Member (Extensions)
|
||||
|
||||
```
|
||||
- membership_fee_type_id (FK → membership_fee_types.id, NOT NULL, default from settings)
|
||||
- membership_fee_start_date (Date, nullable) - When to start generating membership fees
|
||||
- exit_date (Date, nullable) - Exit date (existing)
|
||||
```
|
||||
|
||||
**Logic for membership_fee_start_date:**
|
||||
|
||||
- Auto-set based on global setting `include_joining_cycle`
|
||||
- If `include_joining_cycle = true`: First day of joining month/quarter/year
|
||||
- If `include_joining_cycle = false`: First day of NEXT cycle after joining
|
||||
- Can be manually overridden by admin
|
||||
|
||||
**NO** `include_joining_cycle` field on Member - unnecessary due to `membership_fee_start_date`!
|
||||
|
||||
### Global Settings
|
||||
|
||||
```
|
||||
key: "membership_fees.include_joining_cycle"
|
||||
value: Boolean (Default: true)
|
||||
|
||||
key: "membership_fees.default_membership_fee_type_id"
|
||||
value: UUID (Required) - Default membership fee type for new members
|
||||
```
|
||||
|
||||
**Meaning include_joining_cycle:**
|
||||
|
||||
- `true`: Joining cycle is included (member pays from joining cycle)
|
||||
- `false`: Only from next full cycle after joining
|
||||
|
||||
**Meaning of default membership fee type setting:**
|
||||
|
||||
- Every new member automatically gets this membership fee type
|
||||
- Must be configured in admin settings
|
||||
- Prevents: Members without membership fee type
|
||||
- `membership_fees.include_joining_cycle` — Boolean (default `true`): whether the joining cycle is billed.
|
||||
- `membership_fees.default_membership_fee_type_id` — UUID (required): fee type auto-assigned to every new member; must be configured in admin settings (prevents members without a fee type).
|
||||
|
||||
---
|
||||
|
||||
## Business Logic
|
||||
|
||||
### Cycle Generation
|
||||
### Cycle generation
|
||||
|
||||
**Triggers:**
|
||||
**Triggers:** fee type assigned (incl. at member creation), new cycle begins (cron daily/weekly), admin manual regeneration. Uses PostgreSQL advisory locks per member.
|
||||
|
||||
- Member gets membership fee type assigned (also during member creation)
|
||||
- New cycle begins (Cron job daily/weekly)
|
||||
- Admin requests manual regeneration
|
||||
|
||||
**Algorithm:**
|
||||
|
||||
Use PostgreSQL advisory locks per member to prevent race conditions
|
||||
|
||||
1. Get `member.membership_fee_start_date` and member's membership fee type
|
||||
2. Determine generation start point:
|
||||
- If NO cycles exist: Start from `membership_fee_start_date`
|
||||
- If cycles exist: Start from the cycle AFTER the last existing one
|
||||
3. Generate cycles until today (or `exit_date` if present):
|
||||
- Use the interval to generate the cycles
|
||||
- **Note:** If cycles were explicitly deleted (gaps exist), they are NOT recreated.
|
||||
The generator always continues from the cycle AFTER the last existing cycle.
|
||||
4. Set `amount` to current membership fee type's amount
|
||||
**Algorithm:** start from `membership_fee_start_date` if no cycles exist, else from the cycle AFTER the last existing one; generate to today (or `exit_date`); set each cycle's `amount` from the current fee type. **Deleted cycles (gaps) are NOT recreated** — generation always continues after the last existing cycle. (Full algorithm in architecture doc.)
|
||||
|
||||
**Example (Yearly):**
|
||||
|
||||
|
|
@ -207,93 +104,31 @@ Generated cycles:
|
|||
- ...
|
||||
```
|
||||
|
||||
### Status Transitions
|
||||
### Status transitions
|
||||
|
||||
```
|
||||
unpaid → paid
|
||||
unpaid → suspended
|
||||
paid → unpaid
|
||||
suspended → paid
|
||||
suspended → unpaid
|
||||
```
|
||||
unpaid → paid · unpaid → suspended · paid → unpaid · suspended → paid · suspended → unpaid. Admin + Treasurer (Kassenwart) can change status, via the existing permission system.
|
||||
|
||||
**Permissions:**
|
||||
### Membership fee type change
|
||||
|
||||
- Admin + Treasurer (Kassenwart) can change status
|
||||
- Uses existing permission system
|
||||
MVP allows changing only to a fee type with the **same interval** (e.g. "Regular (yearly)" → "Reduced (yearly)" ✓; → "Reduced (monthly)" ✗). On change: set `member.membership_fee_type_id`; future **unpaid** cycles deleted and regenerated with the new amount; paid/suspended cycles unchanged (historical amount). Future: enable interval switching (overlap handling, extra validation).
|
||||
|
||||
### Membership Fee Type Change
|
||||
### Member exit
|
||||
|
||||
**MVP - Same Cycle Only:**
|
||||
|
||||
- Member can only choose membership fee type with **same cycle**
|
||||
- Example: From "Regular (yearly)" to "Reduced (yearly)" ✓
|
||||
- Example: From "Regular (yearly)" to "Reduced (monthly)" ✗
|
||||
|
||||
**Logic on Change:**
|
||||
|
||||
1. Check: New membership fee type has same interval
|
||||
2. If yes: Set `member.membership_fee_type_id`
|
||||
3. Future **unpaid** cycles: Delete and regenerate with new amount
|
||||
4. Paid/suspended cycles: Remain unchanged (historical amount)
|
||||
|
||||
**Future - Different Intervals:**
|
||||
|
||||
- Enable interval switching (e.g., yearly → monthly)
|
||||
- More complex logic for cycle overlaps
|
||||
- Needs additional validation
|
||||
|
||||
### Member Exit
|
||||
|
||||
**Logic:**
|
||||
|
||||
- Cycles only generated until `member.exit_date`
|
||||
- Existing cycles remain visible
|
||||
- Unpaid exit cycle can be marked as "suspended"
|
||||
|
||||
**Example:**
|
||||
|
||||
```
|
||||
Exit: 15.08.2024
|
||||
Yearly cycle: 01.01.2024 - 31.12.2024
|
||||
|
||||
→ Cycle 2024 is shown (Status: unpaid)
|
||||
→ Admin can set to "suspended"
|
||||
→ No cycles for 2025+ generated
|
||||
```
|
||||
Cycles generated only up to `member.exit_date`; existing cycles remain visible; an unpaid exit cycle can be marked "suspended". E.g. exit 15.08.2024 with a yearly cycle 01.01.–31.12.2024 → 2024 cycle shown (unpaid, admin may suspend); no 2025+ cycles generated.
|
||||
|
||||
---
|
||||
|
||||
## UI/UX Design
|
||||
|
||||
### Member List View
|
||||
### Member List View — column "Membership Fee Status"
|
||||
|
||||
**New Column: "Membership Fee Status"**
|
||||
- **Default (last completed cycle):** in 2024, shows the 2023 status. Color: green = paid ✓, red = unpaid ✗, gray = suspended ⊘.
|
||||
- **Optional toggle:** "Show current cycle" (2024).
|
||||
- **Filters:** "Unpaid membership fees in last cycle", "Unpaid membership fees in current cycle".
|
||||
|
||||
**Default Display (Last Cycle):**
|
||||
### Member Detail View — section "Membership Fees"
|
||||
|
||||
- Shows status of **last completed** cycle
|
||||
- Example in 2024: Shows membership fee for 2023
|
||||
- Color coding:
|
||||
- Green: paid ✓
|
||||
- Red: unpaid ✗
|
||||
- Gray: suspended ⊘
|
||||
|
||||
**Optional: Show Current Cycle**
|
||||
|
||||
- Toggle: "Show current cycle" (2024)
|
||||
- Admin decides what to display
|
||||
|
||||
**Filters:**
|
||||
|
||||
- "Unpaid membership fees in last cycle"
|
||||
- "Unpaid membership fees in current cycle"
|
||||
|
||||
### Member Detail View
|
||||
|
||||
**Section: "Membership Fees"**
|
||||
|
||||
**Membership Fee Type Assignment:**
|
||||
**Fee type assignment:**
|
||||
|
||||
```
|
||||
┌─────────────────────────────────────┐
|
||||
|
|
@ -303,7 +138,7 @@ Yearly cycle: 01.01.2024 - 31.12.2024
|
|||
└─────────────────────────────────────┘
|
||||
```
|
||||
|
||||
**Cycle Table:**
|
||||
**Cycle table:**
|
||||
|
||||
```
|
||||
┌───────────────┬──────────┬────────┬──────────┬─────────┐
|
||||
|
|
@ -322,11 +157,7 @@ Yearly cycle: 01.01.2024 - 31.12.2024
|
|||
Legend: ☑ = paid | ☐ = unpaid | ⊘ = suspended
|
||||
```
|
||||
|
||||
**Quick Marking:**
|
||||
|
||||
- Checkbox in each row for fast marking
|
||||
- Button: "Mark selected as paid/unpaid/suspended"
|
||||
- Bulk action for multiple cycles
|
||||
**Quick marking:** checkbox per row; "Mark selected as paid/unpaid/suspended" button; bulk action for multiple cycles.
|
||||
|
||||
### Admin: Membership Fee Types Management
|
||||
|
||||
|
|
@ -342,14 +173,9 @@ Legend: ☑ = paid | ☐ = unpaid | ⊘ = suspended
|
|||
└────────────┴──────────┴──────────┴────────────┴─────────┘
|
||||
```
|
||||
|
||||
**Edit:**
|
||||
**Edit:** Name ✓, Amount ✓, Description ✓ editable; Interval ✗ **NOT** editable (grayed out).
|
||||
|
||||
- Name: ✓ editable
|
||||
- Amount: ✓ editable
|
||||
- Description: ✓ editable
|
||||
- Interval: ✗ **NOT** editable (grayed out)
|
||||
|
||||
**Warning on Amount Change:**
|
||||
**Warning on amount change:**
|
||||
|
||||
```
|
||||
⚠ Change amount to 65 €?
|
||||
|
|
@ -362,9 +188,7 @@ Impact:
|
|||
[Cancel] [Confirm]
|
||||
```
|
||||
|
||||
### Admin: Settings
|
||||
|
||||
**Membership Fee Configuration:**
|
||||
### Admin: Settings — Membership Fee Configuration
|
||||
|
||||
```
|
||||
Default Membership Fee Type: [Dropdown: Membership Fee Types]
|
||||
|
|
@ -397,135 +221,58 @@ Joining: 15.03.2023
|
|||
|
||||
## Edge Cases
|
||||
|
||||
### 1. Membership Fee Type Change with Different Interval
|
||||
1. **Type change with different interval:** MVP blocks it. UI message:
|
||||
|
||||
**MVP:** Blocked (only same interval allowed)
|
||||
```
|
||||
Error: Interval change not possible
|
||||
|
||||
**UI:**
|
||||
Current membership fee type: "Regular (Yearly)"
|
||||
Selected membership fee type: "Student (Monthly)"
|
||||
|
||||
```
|
||||
Error: Interval change not possible
|
||||
Changing the interval is currently not possible.
|
||||
Please select a membership fee type with interval "Yearly".
|
||||
|
||||
Current membership fee type: "Regular (Yearly)"
|
||||
Selected membership fee type: "Student (Monthly)"
|
||||
[OK]
|
||||
```
|
||||
|
||||
Changing the interval is currently not possible.
|
||||
Please select a membership fee type with interval "Yearly".
|
||||
Future: allow interval switching with overlap calculation and no duplicate cycles.
|
||||
|
||||
[OK]
|
||||
```
|
||||
2. **Exit with unpaid fees (low prio):** on exit, offer to mark unpaid cycles "suspended".
|
||||
|
||||
**Future:**
|
||||
```
|
||||
⚠ Unpaid membership fees present
|
||||
|
||||
- Allow interval switching
|
||||
- Calculate overlaps
|
||||
- Generate new cycles without duplicates
|
||||
This member has 1 unpaid cycle(s):
|
||||
- 2024: 60 € (unpaid)
|
||||
|
||||
### 2. Exit with Unpaid Membership Fees
|
||||
Do you want to continue?
|
||||
|
||||
**Scenario:**
|
||||
[ ] Mark membership fee as "suspended"
|
||||
[Cancel] [Confirm Exit]
|
||||
```
|
||||
|
||||
```
|
||||
Member exits: 15.08.2024
|
||||
Yearly cycle 2024: unpaid
|
||||
```
|
||||
3. **Multiple unpaid cycles:** all shown; select several and bulk-mark.
|
||||
|
||||
**UI Notice on Exit: (Low Prio)**
|
||||
```
|
||||
┌───────────────┬──────────┬────────┬──────────┬─────────┐
|
||||
│ 2023 │ Yearly │ 50 € │ ☐ Open │ [✓] │
|
||||
│ 2024 │ Yearly │ 60 € │ ☐ Open │ [✓] │
|
||||
│ 2025 │ Yearly │ 60 € │ ☐ Open │ [ ] │
|
||||
└───────────────┴──────────┴────────┴──────────┴─────────┘
|
||||
|
||||
```
|
||||
⚠ Unpaid membership fees present
|
||||
[Mark selected as paid/unpaid/suspended] (2 selected)
|
||||
```
|
||||
|
||||
This member has 1 unpaid cycle(s):
|
||||
- 2024: 60 € (unpaid)
|
||||
4. **Amount changes:** 2023 Regular = 50 €, 2024 = 60 € → 2023 cycle keeps 50 € (history), 2024 generated with 60 €; each cycle shows its historical amount.
|
||||
|
||||
Do you want to continue?
|
||||
|
||||
[ ] Mark membership fee as "suspended"
|
||||
[Cancel] [Confirm Exit]
|
||||
```
|
||||
|
||||
### 3. Multiple Unpaid Cycles
|
||||
|
||||
**Scenario:** Member hasn't paid for 2 years
|
||||
|
||||
**Display:**
|
||||
|
||||
```
|
||||
┌───────────────┬──────────┬────────┬──────────┬─────────┐
|
||||
│ 2023 │ Yearly │ 50 € │ ☐ Open │ [✓] │
|
||||
│ 2024 │ Yearly │ 60 € │ ☐ Open │ [✓] │
|
||||
│ 2025 │ Yearly │ 60 € │ ☐ Open │ [ ] │
|
||||
└───────────────┴──────────┴────────┴──────────┴─────────┘
|
||||
|
||||
[Mark selected as paid/unpaid/suspended] (2 selected)
|
||||
```
|
||||
|
||||
### 4. Amount Changes
|
||||
|
||||
**Scenario:**
|
||||
|
||||
```
|
||||
2023: Regular = 50 €
|
||||
2024: Regular = 60 € (increase)
|
||||
```
|
||||
|
||||
**Result:**
|
||||
|
||||
- Cycle 2023: Saved with 50 € (history)
|
||||
- Cycle 2024: Generated with 60 € (current)
|
||||
- Both cycles show correct historical amount
|
||||
|
||||
### 5. Date Boundaries
|
||||
|
||||
**Problem:** What if today = 01.01.2025?
|
||||
|
||||
**Solution:**
|
||||
|
||||
- Current cycle (2025) is generated
|
||||
- Status: unpaid (open)
|
||||
- Shown in overview
|
||||
5. **Date boundaries:** today = 01.01.2025 → current 2025 cycle generated, status unpaid, shown in overview.
|
||||
|
||||
---
|
||||
|
||||
## Implementation Scope
|
||||
|
||||
### MVP (Phase 1)
|
||||
**MVP (Phase 1) — included:** fee types CRUD; automatic cycle generation; status management (paid/unpaid/suspended); member overview with status; per-member cycle view; quick checkbox marking; bulk actions; amount history; same-interval type change; default fee type; joining-cycle configuration.
|
||||
|
||||
**Included:**
|
||||
**NOT included:** interval change; payment details (date, method); automatic vereinfacht.digital integration; prorata; reports/statistics; reminders/dunning (manual via filters).
|
||||
|
||||
- ✓ Membership fee types (CRUD)
|
||||
- ✓ Automatic cycle generation
|
||||
- ✓ Status management (paid/unpaid/suspended)
|
||||
- ✓ Member overview with membership fee status
|
||||
- ✓ Cycle view per member
|
||||
- ✓ Quick checkbox marking
|
||||
- ✓ Bulk actions
|
||||
- ✓ Amount history
|
||||
- ✓ Same-interval type change
|
||||
- ✓ Default membership fee type
|
||||
- ✓ Joining cycle configuration
|
||||
|
||||
**NOT Included:**
|
||||
|
||||
- ✗ Interval change (only same interval)
|
||||
- ✗ Payment details (date, method)
|
||||
- ✗ Automatic integration (vereinfacht.digital)
|
||||
- ✗ Prorata calculation
|
||||
- ✗ Reports/statistics
|
||||
- ✗ Reminders/dunning (manual via filters)
|
||||
|
||||
### Future Enhancements
|
||||
|
||||
**Phase 2:**
|
||||
|
||||
- Payment details (date, amount, method)
|
||||
- Interval change for future unpaid cycles
|
||||
- Manual vereinfacht.digital links per member
|
||||
- Extended filter options
|
||||
|
||||
**Phase 3:**
|
||||
|
||||
- Automated vereinfacht.digital integration
|
||||
- Automatic payment matching
|
||||
- SEPA integration
|
||||
- Advanced reports
|
||||
**Future:** Phase 2 — payment details, interval change for future unpaid cycles, manual vereinfacht.digital links per member, extended filters. Phase 3 — automated vereinfacht.digital integration, automatic payment matching, SEPA, advanced reports.
|
||||
|
|
|
|||
|
|
@ -1,20 +1,16 @@
|
|||
# Onboarding & Join – High-Level Concept
|
||||
|
||||
**Status:** Draft for design decisions and implementation specs. **Prio 1 (Subtasks 1–4) implemented.**
|
||||
**Scope:** Prio 1 = public Join form; Step 2 = Vorstand approval. Invite-Link and OIDC JIT are out of scope and documented only as future entry paths.
|
||||
**Status:** Prio 1 (Subtasks 1–4) and Step 2 (Vorstand approval, Subtask 5) implemented. The Invite-Link / OIDC-JIT join entry paths (§4) are designed here but **not yet implemented**.
|
||||
**Related:** Issue #308, roles-and-permissions-architecture, page-permission-route-coverage.
|
||||
|
||||
---
|
||||
|
||||
## 1. Focus and Goals
|
||||
|
||||
- **Focus:** Onboarding and **initial data capture**, not self-service editing of existing members.
|
||||
- **Entry paths (vision):**
|
||||
- **Public Join form** (Prio 1) – unauthenticated submission.
|
||||
- **Invite link** (tokenized) – later.
|
||||
- **OIDC first-login** (Just-in-Time Provisioning) – later.
|
||||
- **Admin control:** All entry paths and their behaviour (e.g. which fields, approval required) shall be configurable by admins; MVP can start with sensible defaults.
|
||||
- **Approval:** A Vorstand (board) approval step is a direct follow-up (Step 2) after the public Join; the data model and flow must support it.
|
||||
- **Focus:** onboarding and **initial data capture**, not self-service editing of existing members.
|
||||
- **Entry paths (vision):** public Join form (Prio 1, unauthenticated submission); invite link (tokenized, later); OIDC first-login / Just-in-Time Provisioning (later).
|
||||
- **Admin control:** all entry paths and their behaviour (which fields, approval required) shall be admin-configurable; MVP may start with sensible defaults.
|
||||
- **Approval:** a Vorstand (board) approval step is the direct follow-up (Step 2) after the public Join; the data model and flow support it.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -22,168 +18,137 @@
|
|||
|
||||
### 2.1 Intent
|
||||
|
||||
- **Public** page (e.g. `/join`): no login; anyone can open and submit.
|
||||
- Result is **not** a User or Member. Result is an **onboarding / join request**: the JoinRequest record is **created in the database on form submit** in status `pending_confirmation`, then **updated to** `submitted` after the user clicks the confirmation link.
|
||||
- This keeps:
|
||||
- **Public intake** (abuse-prone) separate from **identity and account creation** (after approval / invite / OIDC).
|
||||
- Existing policies (e.g. User–Member linking, admin-only link) untouched until a defined "promotion" flow (e.g. after approval) creates User/Member.
|
||||
- **Elixir/Phoenix/Ash standard:** Data is persisted in the database from the start (one Ash resource, status-driven flow). No ETS or stateless token for pre-confirmation storage; confirm flow only updates the existing record.
|
||||
- **Public** page `/join`: no login; anyone can open and submit.
|
||||
- The result is **not** a User or Member but a **JoinRequest** record, created in the DB on form submit in status `pending_confirmation`, then updated to `submitted` after the user clicks the confirmation link.
|
||||
- This keeps public intake (abuse-prone) separate from identity/account creation, and leaves existing policies (User–Member linking, admin-only link) untouched until a defined promotion flow (after approval) creates User/Member.
|
||||
- **Standard:** data is persisted in the DB from the start (one Ash resource, status-driven). No ETS or stateless token for pre-confirmation storage; the confirm flow only updates the existing record.
|
||||
|
||||
### 2.2 User Flow (Prio 1)
|
||||
### 2.2 User Flow
|
||||
|
||||
1. Unauthenticated user opens `/join`.
|
||||
2. Short explanation + form (what happens next: "We will review … you will hear from us").
|
||||
3. **Submit** → A **JoinRequest is created** in the database with status `pending_confirmation`; confirmation email is sent; user sees: "We have saved your details. To complete your request, please click the link we sent to your email."
|
||||
4. **User clicks confirmation link** → The existing JoinRequest is **updated** to status `submitted` (`submitted_at` set, confirmation token invalidated); user sees: "Thank you, we have received your request."
|
||||
2. Short explanation + form ("We will review … you will hear from us").
|
||||
3. **Submit** → JoinRequest created with status `pending_confirmation`; confirmation email sent; user sees "We have saved your details. To complete your request, please click the link we sent to your email."
|
||||
4. **User clicks confirmation link** → existing JoinRequest updated to `submitted` (`submitted_at` set, confirmation token invalidated); user sees "Thank you, we have received your request."
|
||||
|
||||
**Rationale (double opt-in with DB-first):** Email confirmation remains best practice (we only treat the request as "submitted" after the link is clicked). The record exists in the DB from submit time so we use standard Phoenix/Ash persistence, multi-node safety, and a simple status transition (`pending_confirmation` → `submitted`) on confirm. This aligns with patterns like AshAuthentication (resource exists before confirm; confirm updates state).
|
||||
|
||||
**Out of scope for Prio 1:** Approval UI, account creation, OIDC, invite links.
|
||||
**Rationale (double opt-in, DB-first):** email confirmation stays best practice (treated as "submitted" only after the click); the record exists in the DB from submit time, so we get standard Phoenix/Ash persistence, multi-node safety, and a simple `pending_confirmation → submitted` transition. Aligns with AshAuthentication (resource exists before confirm; confirm updates state).
|
||||
|
||||
### 2.3 Data Flow
|
||||
|
||||
- **Input:** Only data explicitly allowed for the public form; field set is admin-configured (see §2.6). No internal or sensitive fields. **Server-side allowlist:** The set of accepted fields is enforced in the LiveView (`build_submit_attrs`) and in the resource via **`JoinRequest.Changes.FilterFormDataByAllowlist`** so that even direct API/submit_join_request calls only persist allowlisted form_data keys.
|
||||
- **On form submit:** **Create** a JoinRequest with status `pending_confirmation`, store confirmation token **hash** in the DB (raw token only in the email link), set `confirmation_token_expires_at` (e.g. 24h), store all allowlisted form data (see §2.3.2), then send confirmation email.
|
||||
- **On confirmation link click:** **Update** the JoinRequest (find by token hash): set status to `submitted`, set `submitted_at`, clear/invalidate token fields. If the record is already `submitted`, return success without changing it (idempotent).
|
||||
- **No creation** of Member or User in Prio 1; promotion to Member/User happens in a later step (e.g. after approval).
|
||||
- **Input:** only data explicitly allowed for the public form; field set is admin-configured (§2.6). No internal/sensitive fields. **Server-side allowlist:** accepted fields are enforced both in the LiveView (`build_submit_attrs`) and in the resource via **`JoinRequest.Changes.FilterFormDataByAllowlist`**, so even direct API / `submit_join_request` calls persist only allowlisted `form_data` keys.
|
||||
- **On submit:** create a JoinRequest (status `pending_confirmation`), store confirmation token **hash** in the DB (raw token only in the email link), set `confirmation_token_expires_at` (e.g. 24h), store all allowlisted form data, then send the confirmation email.
|
||||
- **On confirm link click:** find by token hash, set status `submitted`, set `submitted_at`, clear/invalidate token fields. If already `submitted`, return success without changing it (idempotent).
|
||||
- **No Member/User creation** in Prio 1; promotion happens later (after approval).
|
||||
|
||||
#### 2.3.1 Pre-Confirmation Store (Decided)
|
||||
|
||||
**Decision:** Store in the **database** only. Use the **same** JoinRequest resource and table from the start.
|
||||
**Decision:** store in the **database** only, using the **same** JoinRequest resource and table throughout. On submit, create one row (`pending_confirmation`, token hash, expiry); on confirm, update that row to `submitted` — no second table, no ETS, no stateless token.
|
||||
|
||||
- On submit: **create** one JoinRequest row with status `pending_confirmation`, confirmation token **hash**, and expiry.
|
||||
- On confirm: **update** that row to status `submitted` (no second table, no ETS, no stateless token).
|
||||
- **Retention and cleanup:** JoinRequests that remain in `pending_confirmation` past the token expiry (e.g. 24 hours) are **hard-deleted** by a scheduled job (e.g. Oban cron). Retention period: **24 hours**; document in DSGVO/retention as needed.
|
||||
- **Rationale:** Elixir/Phoenix/Ash standard is persistence in DB, one resource, status machine. Multi-node safe, restart safe, and cleanup is a standard cron task.
|
||||
**Retention and cleanup:** JoinRequests still in `pending_confirmation` past the token expiry are **hard-deleted** by a scheduled job (Oban cron). Retention period: **24 hours**; document in DSGVO/retention as needed. Multi-node and restart safe; cleanup is a standard cron task.
|
||||
|
||||
#### 2.3.2 JoinRequest: Data Model and Schema
|
||||
|
||||
- **Status:** `pending_confirmation` (initial, after form submit) → `submitted` (after link click) → later `approved` / `rejected`. Include **approved_at**, **rejected_at**, **reviewed_by_user_id** for audit.
|
||||
- **Confirmation:** Store **confirmation_token_hash** (not the raw token); **confirmation_token_expires_at**; optional **confirmation_sent_at**. Raw token appears only in the email link; on confirm, hash the incoming token and find the record by hash.
|
||||
- **Payload vs typed columns (recommendation):**
|
||||
- **Typed columns** for **email** (required, dedicated field for index, search, dedup, audit) and for **first_name** and **last_name** (optional). These align with `Mv.Constants.member_fields()` and with the existing Member resource; they support approval-list display and straightforward promotion to Member without parsing JSON.
|
||||
- **Remaining form data** (other member fields + custom field values) in a **jsonb** attribute (e.g. `form_data`) plus a **schema_version** (e.g. tied to join-form or member_fields evolution) so future changes do not break existing records.
|
||||
- **What it depends on:** (1) Whether the join form field set is fixed or often extended – if fixed, more typed columns are feasible; if very dynamic, keeping the rest in jsonb avoids migrations. (2) Whether the approval UI or reporting needs to filter/sort by other fields (e.g. city) – if yes, consider adding those as typed columns later. For MVP, email + first_name + last_name typed and rest in jsonb is a good balance with the current codebase (Member has typed attributes; export/import use allowlists of field names).
|
||||
- **Logger hygiene:** Do not log the full payload/form_data; follow CODE_GUIDELINES on log sanitization.
|
||||
- **Idempotency:** Confirm action finds the JoinRequest by token hash; if status is already `submitted`, return success without updating. Optionally enforce **unique_index on confirmation_token_hash** so the same token cannot apply to more than one record.
|
||||
- **Abuse metadata:** If stored (e.g. IP hash), classify as **security telemetry** or **personally identifiable** (DSGVO). Prefer hashed/aggregated values only (e.g. /24 prefix hash or keyed-hash), not raw IP; document classification and retention. Out of scope for Prio 1 unless explicitly added.
|
||||
- **Status:** `pending_confirmation` (initial) → `submitted` (after link click) → later `approved` / `rejected`. Audit: **approved_at**, **rejected_at**, **reviewed_by_user_id**.
|
||||
- **Confirmation:** store **confirmation_token_hash** (not the raw token); **confirmation_token_expires_at**; optional **confirmation_sent_at**. The raw token appears only in the email link; on confirm, hash the incoming token and find the record by hash.
|
||||
- **Payload vs typed columns:** **typed columns** for **email** (required — dedicated field for index, search, dedup, audit) and **first_name** / **last_name** (optional); these align with `Mv.Constants.member_fields()` and the Member resource, supporting approval-list display and straightforward promotion without parsing JSON. **Remaining form data** (other member fields + custom field values) goes in a **jsonb** attribute (`form_data`) plus a **schema_version** so future changes don't break existing records.
|
||||
- *Depends on:* (1) whether the join-form field set is fixed (more typed columns feasible) or dynamic (keep rest in jsonb to avoid migrations); (2) whether approval UI/reporting needs to filter/sort by other fields (e.g. city) — if so, add typed columns later. For MVP, email + first_name + last_name typed and the rest in jsonb balances well with the current codebase.
|
||||
- **Logger hygiene:** do not log the full payload/`form_data`; follow CODE_GUIDELINES on log sanitization.
|
||||
- **Idempotency:** confirm finds the JoinRequest by token hash; if already `submitted`, return success without updating. Optionally enforce a **unique_index on confirmation_token_hash**.
|
||||
- **Abuse metadata:** if stored (e.g. IP hash), classify as security telemetry or PII (DSGVO). Prefer hashed/aggregated values only (e.g. /24 prefix hash or keyed-hash), not raw IP; document classification and retention. Out of scope for Prio 1 unless explicitly added.
|
||||
|
||||
### 2.4 Security
|
||||
|
||||
- **Public paths:** `/join` and the confirmation route must be public (unauthenticated access returns 200).
|
||||
- **Explicit public path for `/join`:** Add **`/join`** (and if needed `/join/*`) to the page-permission plug’s **`public_path?/1`** so that the join page is reachable without login. Do not rely on the confirm path alone.
|
||||
- **Confirmation route:** Use **`/confirm_join/:token`** so that the existing whitelist (e.g. `String.starts_with?(path, "/confirm")`) already covers it; no extra plug change for confirm.
|
||||
- **Abuse:** **Honeypot** (MVP) plus **rate limiting** (MVP). Use Phoenix/Elixir standard options (e.g. **Hammer** with **Hammer.Plug**, ETS backend), scoped to the join flow (e.g. by IP). Client IP for rate limiting: prefer **X-Forwarded-For** / **X-Real-IP** when behind a reverse proxy (see Endpoint `connect_info: [:x_headers]` and `JoinLive.client_ip_from_socket/1`); fallback to peer_data with correct IPv4/IPv6 string via `:inet.ntoa/1`. Verify library version and multi-node behaviour before or during implementation.
|
||||
- **Data:** Minimal PII; no sensitive data on the public form; consider DSGVO when extending. If abuse signals are stored: only hashed or aggregated values; document classification and retention.
|
||||
- **Approval-only:** No automatic User/Member creation from the join form; approval (Step 2) or other trusted path creates identity.
|
||||
- **Ash policies and actor:** JoinRequest has **explicit public actions** allowed with `actor: nil` (e.g. `submit` for create, `confirm` for update). Model via **policies** that permit these actions when actor is nil; do **not** use `authorize?: false` unless documented and clearly not a privilege-escalation path.
|
||||
- **No system-actor fallback:** Join and confirmation run without an authenticated user. Do **not** use the system actor as a fallback for "missing actor". Use explicit unauthenticated context; see CODE_GUIDELINES §5.0.
|
||||
- **Explicit public path for `/join`:** add **`/join`** (and if needed `/join/*`) to the page-permission plug's **`public_path?/1`**; do not rely on the confirm path alone.
|
||||
- **Confirmation route:** use **`/confirm_join/:token`** so the existing whitelist (e.g. `String.starts_with?(path, "/confirm")`) already covers it — no extra plug change for confirm.
|
||||
- **Abuse:** **honeypot** + **rate limiting** in MVP (e.g. **Hammer** with **Hammer.Plug**, ETS backend), scoped to the join flow (e.g. by IP). Client IP: prefer **X-Forwarded-For** / **X-Real-IP** behind a reverse proxy (Endpoint `connect_info: [:x_headers]`, `JoinLive.client_ip_from_socket/1`); fallback to peer_data with correct IPv4/IPv6 string via `:inet.ntoa/1`. Verify library version and multi-node behaviour.
|
||||
- **Data:** minimal PII; no sensitive data on the public form; consider DSGVO when extending. Stored abuse signals: only hashed/aggregated, documented.
|
||||
- **Approval-only:** no automatic User/Member creation from the join form; approval (Step 2) or another trusted path creates identity.
|
||||
- **Ash policies and actor:** JoinRequest has **explicit public actions** allowed with `actor: nil` (`submit` for create, `confirm` for update). Model via **policies** that permit these actions when actor is nil; do **not** use `authorize?: false` unless documented and clearly not a privilege-escalation path.
|
||||
- **No system-actor fallback:** join and confirmation run without an authenticated user. Do **not** use the system actor as a fallback for a "missing actor"; use an explicit unauthenticated context. See CODE_GUIDELINES §5.0 and `lib/mv/authorization/checks/actor_is_nil.ex`.
|
||||
|
||||
### 2.5 Usability and UX
|
||||
|
||||
- **After submit:** Communicate clearly: e.g. "We have saved your details. To complete your request, please click the link we sent to your email." (Exact copy in implementation spec.)
|
||||
- Clear heading and short copy (e.g. "Become a member / Submit request" and "What happens next").
|
||||
- **After submit:** "We have saved your details. To complete your request, please click the link we sent to your email."
|
||||
- Clear heading + short copy ("Become a member / Submit request", "What happens next").
|
||||
- Form only as simple as needed (conversion vs. data hunger).
|
||||
- Success message after confirm: neutral, no promise of an account ("We will get in touch").
|
||||
- **Expired confirmation link:** If the user clicks after the token has expired, show a clear message (e.g. "This link has expired") and instruct them to submit the form again. Specify exact copy and behaviour in the implementation spec.
|
||||
- **Re-send confirmation link:** Out of scope for Prio 1. If not implemented in Prio 1, **create a separate ticket immediately**. Example UX: "Request new confirmation email" on the "Please confirm your email" or expired-link page.
|
||||
- Accessibility and i18n: same standards as rest of the app (labels, errors, Gettext).
|
||||
- Confirm success message: neutral, no promise of an account ("We will get in touch").
|
||||
- **Expired confirmation link:** clear message ("This link has expired") + instruction to submit the form again. Exact copy in the implementation spec.
|
||||
- **Re-send confirmation link:** out of scope for Prio 1; if not implemented, **create a separate ticket immediately**. Example UX: "Request new confirmation email" on the confirm/expired page.
|
||||
- Accessibility and i18n: same standards as the rest of the app (labels, errors, Gettext).
|
||||
|
||||
### 2.6 Admin Configurability: Join Form Settings
|
||||
|
||||
- **Placement:** Own section **"Onboarding / Join"** in global settings, **above** "Custom fields", **below** "Vereinsdaten" (club data).
|
||||
- **Join form enabled:** Checkbox (e.g. `join_form_enabled`). When set, the public `/join` page is active and the following config applies.
|
||||
- **Copyable join link:** When the join form is enabled, a copyable full URL to the `/join` page is shown below the checkbox (above the field list), with a short hint so admins can share it with applicants.
|
||||
- **Field selection:** From **all existing** member fields (from `Mv.Constants.member_fields()`) and **custom fields**, the admin selects which fields appear on the join form. Stored as a list/set of field identifiers (no separate table); display in settings as a simple list, e.g. **badges with X to remove** (similar to the groups overview). Adding fields: e.g. dropdown or modal to pick from remaining fields. Detailed UX for this subsection is to be specified in a **separate subtask**.
|
||||
- **Technically required fields:** The only field that must always be required for the join flow is **email**. All other fields can be optional or marked as required per admin choice; implementation should support a "required" flag per selected join-form field.
|
||||
- **Other:** Which entry paths are enabled, approval workflow (who can approve) – to be detailed in Step 2 and later specs.
|
||||
- **Placement:** own section **"Onboarding / Join"** in global settings, **above** "Custom fields", **below** "Vereinsdaten".
|
||||
- **Join form enabled:** checkbox (`join_form_enabled`); when set, the public `/join` page is active and the config below applies.
|
||||
- **Copyable join link:** when enabled, a copyable full URL to `/join` is shown below the checkbox (above the field list), with a short hint for sharing with applicants.
|
||||
- **Field selection:** from **all existing** member fields (`Mv.Constants.member_fields()`) and **custom fields**, the admin picks which appear on the join form. Stored as a list/set of field identifiers (no separate table); displayed as **badges with X to remove** (like the groups overview), added via dropdown/modal. Detailed UX to be specified in a separate subtask.
|
||||
- **Technically required fields:** only **email** must always be required. All others can be optional or marked required per admin choice; support a "required" flag per selected field.
|
||||
- **Other:** which entry paths are enabled, approval workflow (who can approve) — detailed in Step 2 and later specs.
|
||||
|
||||
---
|
||||
|
||||
## 3. Step 2: Vorstand Approval
|
||||
## 3. Step 2: Vorstand Approval (implemented)
|
||||
|
||||
- **Goal:** Board (Vorstand) can review join requests (e.g. list status "submitted") and approve or reject.
|
||||
- **Route:** **`/join_requests`** for the approval UI (list and detail). See §3.1 for full specification.
|
||||
- **Outcome of approval (admin-configurable):**
|
||||
- **Default:** Approval creates **Member only**; no User is created. An admin can link a User later if needed.
|
||||
- **Optional (configurable):** If an option is set, approval may also create a **User** (e.g. invite-to-set-password). This is **open for later**; implementation concepts will be detailed when that option is implemented.
|
||||
- **Permissions:** Approval uses the existing permission set **normal_user** (e.g. role "Kassenwart"). JoinRequest gets read and update (or dedicated approve/reject actions) for scope :all in normal_user, and **`/join_requests`** (and **`/join_requests/:id`** for detail) are added to normal_user’s allowed pages.
|
||||
- **Goal:** the board can review join requests (e.g. list status "submitted") and approve or reject.
|
||||
- **Route:** **`/join_requests`** (list) and **`/join_requests/:id`** (detail), defined in `MvWeb.Router` → `MvWeb.JoinRequestLive.Index` / `.Show`. Full spec in §3.1.
|
||||
- **Outcome of approval:** approval creates a **Member only** (no User; an admin can link a User later). The optional "also create a User on approval" variant is **not yet implemented**.
|
||||
- **Permissions:** approval uses the existing **normal_user** permission set (e.g. role "Kassenwart"). In `Mv.Authorization.PermissionSets`, normal_user has JoinRequest read + update for scope :all, and `/join_requests` and `/join_requests/:id` are in its allowed pages.
|
||||
|
||||
### 3.1 Step 2 – Approval (detail)
|
||||
### 3.1 Step 2 – Approval (detail) — implemented in Subtask 5
|
||||
|
||||
Implementation spec for Subtask 5.
|
||||
**Route and pages:**
|
||||
|
||||
#### Route and pages
|
||||
- **List `/join_requests`:** filter by status (default/primary view: `submitted`); optional view for "all" or "approved/rejected" for audit.
|
||||
- **Detail `/join_requests/:id`:** two blocks — (1) **Applicant data**: all form fields (typed + `form_data`) merged and shown in join-form order; (2) **Status and review**: submitted_at, status, and when decided approved_at/rejected_at + reviewed by. Approve / Reject actions when status is `submitted`.
|
||||
|
||||
- **List:** **`/join_requests`** – list of join requests. Filter by status (default or primary view: status `submitted`); optional view for "all" or "approved/rejected" for audit.
|
||||
- **Detail:** **`/join_requests/:id`** – single join request. **Two blocks:** (1) **Applicant data** – all form fields (typed + `form_data`) merged and shown in join-form order; (2) **Status and review** – submitted_at, status, and when decided: approved_at/rejected_at, reviewed by. Actions Approve / Reject when status is `submitted`.
|
||||
**Backend (`Mv.Membership.JoinRequest`) — actions (authenticated only):**
|
||||
|
||||
#### Backend (JoinRequest)
|
||||
- **`approve`** (update, change `JoinRequest.Changes.ApproveRequest`): allowed only when status is `submitted`. Sets `approved`, `approved_at`, `reviewed_by_user_id` / `reviewed_by_display` (actor). Promotion to Member is driven by the domain function (see below), not the change.
|
||||
- **`reject`** (update, change `JoinRequest.Changes.RejectRequest`): allowed only when status is `submitted`. Sets `rejected`, `rejected_at`, `reviewed_by_user_id`. No reason field in MVP.
|
||||
- **Policies:** `approve` and `reject` are each permitted via **`HasPermission`**; the read policy uses **`HasJoinRequestAccess`** (a SimpleCheck) so list/detail can load data. Not allowed for `actor: nil`.
|
||||
- **Domain (`Mv.Membership`):** `list_join_requests/1` (filter by status, with actor), `approve_join_request/2` (id, actor), `reject_join_request/2` (id, actor).
|
||||
|
||||
- **New actions (authenticated only):**
|
||||
- **`approve`** (update): allowed only when status is `submitted`. Sets status `approved`, `approved_at`, `reviewed_by_user_id` (actor). Triggers promotion to Member (see Promotion below).
|
||||
- **`reject`** (update): allowed only when status is `submitted`. Sets status `rejected`, `rejected_at`, `reviewed_by_user_id`. No reason field in MVP.
|
||||
- **Policies:** `approve` and `reject` permitted via **HasPermission** for permission set **normal_user** (read/update or explicit approve/reject on JoinRequest, scope :all). Not allowed for `actor: nil`.
|
||||
- **Domain:** Expose `list_join_requests/1` (e.g. filter by status, with actor), `approve_join_request/2` (id, actor), `reject_join_request/2` (id, actor). Read action for JoinRequest for normal_user scope :all so list/detail can load data.
|
||||
**Promotion: JoinRequest → Member:**
|
||||
|
||||
#### Promotion: JoinRequest → Member
|
||||
- **When:** on successful `approve` only (status was `submitted`).
|
||||
- **Mapping:** typed fields **email**, **first_name**, **last_name** → Member attributes. **form_data** keys matching `Mv.Constants.member_fields()` (string form) → Member attributes; keys that are custom field IDs (UUID) → **CustomFieldValue** records linked to the new Member.
|
||||
- **Defaults:** `join_date` = today. `membership_fee_type_id` is not set here; the Member `create_member` action applies the default fee type from settings (see `Mv.Membership.Member.Changes.SetDefaultMembershipFeeType`).
|
||||
- **Implementation:** the domain function `Mv.Membership.approve_join_request/2` calls the private `promote_to_member/2`, which builds member attributes + custom_field_values and calls Member `create_member` with the reviewer as actor. No User created in MVP.
|
||||
- **Atomicity:** the approve flow (get → update to approved → promote to Member) runs inside **`Ash.transact(JoinRequest, fn -> ... end)`**, so if Member creation fails (validation, unique constraint) the JoinRequest status rolls back.
|
||||
- **Idempotency:** `ApproveRequest` only transitions from `submitted`; a repeated approve on an already-`approved` request is rejected with a status error, so no duplicate Member is created.
|
||||
|
||||
- **When:** On successful `approve` only (status was `submitted`).
|
||||
- **Mapping:**
|
||||
- JoinRequest typed fields → Member: **email**, **first_name**, **last_name** copied to Member attributes.
|
||||
- **form_data** (jsonb): keys that match `Mv.Constants.member_fields()` (atom names or string keys) → corresponding Member attributes. Keys that are custom field IDs (UUID format) → create **CustomFieldValue** records linked to the new Member.
|
||||
- **Defaults:** e.g. `join_date` = today if not in form_data; `membership_fee_type_id` = default from settings (or first available) if not provided. Handle required Member validations (e.g. email already present from JoinRequest).
|
||||
- **Implementation:** Prefer a single Ash change (e.g. `JoinRequest.Changes.PromoteToMember`) or a domain function that builds member attributes + custom_field_values from the approved JoinRequest and calls Member `create_member` (actor: reviewer or system actor as per CODE_GUIDELINES; document choice). No User created in MVP.
|
||||
- **Atomicity:** The approve flow (get JoinRequest → update to approved → promote to Member) runs inside **`Ash.transact(JoinRequest, fn -> ... end)`** so that if Member creation fails (e.g. validation, unique constraint), the JoinRequest status is rolled back and remains consistent.
|
||||
- **Idempotency:** If approve is called again by mistake (e.g. race), either reject transition when status is already `approved` or ensure Member creation is idempotent (e.g. do not create duplicate Member for same JoinRequest).
|
||||
**Permission sets and routing:**
|
||||
|
||||
#### Permission sets and routing
|
||||
- **PermissionSets (`Mv.Authorization.PermissionSets`, normal_user):** JoinRequest **read** :all and **update** :all; pages `/join_requests` and `/join_requests/:id`.
|
||||
- **Router (`MvWeb.Router`):** live routes `/join_requests` → `JoinRequestLive.Index` and `/join_requests/:id` → `JoinRequestLive.Show`; entries recorded in **page-permission-route-coverage.md**; plug coverage so normal_user is allowed, read_only/own_data denied.
|
||||
|
||||
- **PermissionSets (normal_user):** Add JoinRequest **read** :all and **update** :all (or **approve** / **reject** if using dedicated actions). Add pages **`/join_requests`** and **`/join_requests/:id`** to the normal_user pages list.
|
||||
- **Router:** Register live routes for list and detail; add entries to **page-permission-route-coverage.md** and extend plug tests so normal_user is allowed, read_only/own_data denied.
|
||||
**UI/UX (approval):**
|
||||
|
||||
#### UI/UX (approval)
|
||||
- **List:** table/card list with columns e.g. submitted_at, first_name, last_name, email, status; primary/default filter status = `submitted`; links to detail. Follow existing list patterns (Members/Groups): header, back link, CoreComponents table.
|
||||
- **Detail:** all request data (typed + form_data rendered by field); buttons **Approve** (primary), **Reject** (secondary); reject in MVP has no reason field. Same accessibility/i18n standards.
|
||||
|
||||
- **List:** Table or card list with columns e.g. submitted_at, first_name, last_name, email, status. Primary filter or default filter: status = `submitted`. Buttons or links to open detail. Follow existing list patterns (e.g. Members/Groups): header, back link, CoreComponents table.
|
||||
- **Detail:** Show all request data (typed + form_data rendered by field). Buttons: **Approve** (primary), **Reject** (secondary). Reject in MVP: no reason field; just set status to rejected and audit fields.
|
||||
- **Accessibility and i18n:** Same standards as rest of app (labels, Gettext, keyboard, ARIA where needed).
|
||||
|
||||
#### Tests
|
||||
|
||||
- JoinRequest: policy tests – approve/reject allowed for normal_user (and admin), forbidden for nil/own_data/read_only.
|
||||
- Domain: approve creates one Member with correct mapped data; reject only updates status and audit fields; approve when already approved is no-op or error.
|
||||
- Page permission: normal_user can GET `/join_requests` and `/join_requests/:id`; read_only/own_data cannot.
|
||||
- Optional: LiveView smoke test – list loads, approve/reject from detail updates state.
|
||||
**Tests:** policy tests (approve/reject allowed for normal_user and admin, forbidden for nil/own_data/read_only); domain (approve creates one Member with correct mapped data; reject only updates status + audit; approve-when-already-approved is no-op or error); page permission (normal_user can GET both routes; read_only/own_data cannot); optional LiveView smoke test.
|
||||
|
||||
---
|
||||
|
||||
## 4. Future Entry Paths (Out of Scope Here)
|
||||
## 4. Future Entry Paths (Out of Scope Here, not yet implemented)
|
||||
|
||||
- **Invite link (tokenized):** Unique link per invitee; submission or account creation tied to token.
|
||||
- **OIDC first-login (JIT):** First login via OIDC creates/links User and optionally Member from IdP data.
|
||||
- Both must be design-ready so they can attach to the same approval or creation pipeline later.
|
||||
- **Invite link (tokenized):** unique link per invitee; submission or account creation tied to the token.
|
||||
- **OIDC first-login (JIT):** first OIDC login creates/links a User and optionally a Member from IdP data.
|
||||
- Both must be design-ready so they can attach to the same approval/creation pipeline later.
|
||||
|
||||
---
|
||||
|
||||
## 5. Evaluation of the Proposed Concept Draft
|
||||
## 5. Concept Evaluation — adopted decisions
|
||||
|
||||
**Adopted and reflected above:**
|
||||
- **Naming:** resource **JoinRequest** (one resource, status + audit timestamps).
|
||||
- **No User/Member from `/join`:** only a JoinRequest, created on submit (`pending_confirmation`), updated to `submitted` on confirmation. Member/User domain unchanged.
|
||||
- **Public actions:** `submit` (create with `pending_confirmation` + send email) and `confirm` (update to `submitted`).
|
||||
- **Public paths:** `/join` explicitly added to the plug's public path list; `/confirm_join/:token` covered by the existing `/confirm*` rule.
|
||||
- **Minimal data:** email technically required; other fields from the admin-configured set, with optional "required" per field.
|
||||
- **Security:** honeypot + rate limiting in MVP; email confirmation before "submitted"; token stored as hash; 24h retention + hard-delete for expired pending.
|
||||
|
||||
- **Naming:** Resource name **JoinRequest** (one resource, status + audit timestamps).
|
||||
- **No User/Member from `/join`:** Only a JoinRequest; record is **created on form submit** (status `pending_confirmation`) and **updated to** `submitted` on confirmation. Abuse surface and policy complexity stay low.
|
||||
- **Dedicated resource and actions:** New resource `JoinRequest` with public actions: **submit** (create with `pending_confirmation` + send email) and **confirm** (update to `submitted`). Member/User domain unchanged.
|
||||
- **Public paths:** `/join` is **explicitly** added to the page-permission plug’s public path list; confirmation route `/confirm_join/:token` is covered by existing `/confirm*` rule.
|
||||
- **Minimal data:** Email is technically required; other fields from admin-configured join-form field set, with optional "required" per field.
|
||||
- **Security:** Honeypot + rate limiting in MVP; email confirmation before treating request as submitted; token stored as hash; 24h retention and hard-delete for expired pending.
|
||||
- **Tests:** Unauthenticated GET `/join` → 200; submit creates one JoinRequest (`pending_confirmation`); confirm updates it to `submitted`; idempotent confirm; honeypot and rate limiting covered; public-path tests updated.
|
||||
|
||||
**Refinements in this document:**
|
||||
|
||||
- Approval as Step 2; User creation after approval left open for later.
|
||||
- Admin configurability: join form settings as own section; detailed UX in a subtask.
|
||||
- Three entry paths (public, invite, OIDC) and their place in the roadmap made explicit.
|
||||
- Pre-confirmation store: DB only, one resource, 24h retention, hard-delete.
|
||||
- Payload: typed email (required), first_name, last_name; rest in jsonb with schema_version; rationale and what it depends on documented.
|
||||
Refinements layered in this document: approval as Step 2 (User creation after approval left open); join-form settings as their own section (detailed UX in a subtask); three entry paths placed in the roadmap; pre-confirmation store DB-only with 24h hard-delete; payload split typed (email/first_name/last_name) + jsonb with schema_version.
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -191,90 +156,60 @@ Implementation spec for Subtask 5.
|
|||
|
||||
**Decided:**
|
||||
|
||||
- **Email confirmation (double opt-in):** JoinRequest is **created on form submit** with status `pending_confirmation` and **updated to** `submitted` when the user clicks the confirmation link. Double opt-in is preserved (we only treat as "submitted" after the link is clicked). Existing confirmation pattern (AshAuthentication) is reused for token + email sender + route.
|
||||
- **Email confirmation (double opt-in):** JoinRequest created on submit (`pending_confirmation`), updated to `submitted` on link click; treated as submitted only after the click. Reuses the existing AshAuthentication pattern (token + email sender + route).
|
||||
- **Naming:** **JoinRequest**.
|
||||
- **Pre-confirmation store:** **DB only.** Same JoinRequest resource; no ETS, no stateless token. Confirmation token stored as **hash** in DB; raw token only in email link. **24h** retention for `pending_confirmation`; **hard-delete** of expired records via scheduled job (e.g. Oban cron).
|
||||
- **Confirmation route:** **`/confirm_join/:token`** so existing `starts_with?(path, "/confirm")` covers it.
|
||||
- **Public path for `/join`:** **Add `/join` explicitly** to the page-permission plug’s `public_path?/1` (e.g. in `CheckPagePermission`) so unauthenticated users can reach the join page.
|
||||
- **JoinRequest schema:** Status `pending_confirmation` | `submitted` | `approved` | `rejected`. Typed: **email** (required), **first_name**, **last_name** (optional). **form_data** (jsonb) + **schema_version** for remaining form fields. **confirmation_token_hash**, **confirmation_token_expires_at**; **submitted_at**, **approved_at**, **rejected_at**, **reviewed_by_user_id**, **reviewed_by_display** (denormalized reviewer email for "Geprüft von" without loading User) for audit. Idempotent confirm (unique constraint on token hash or update only when status is `pending_confirmation`).
|
||||
- **Approval outcome:** Admin-configurable. Default: approval creates Member only (no User). Optional "create User on approval" is **left for later**.
|
||||
- **Rate limiting:** Honeypot + rate limiting from the start (e.g. Hammer.Plug).
|
||||
- **Settings:** Own section "Onboarding / Join" in global settings; `join_form_enabled` plus field selection; display as list/badges; detailed UX in a **separate subtask**.
|
||||
- **Approval permission:** normal_user; JoinRequest read/update and approval page added to normal_user; no new permission set.
|
||||
- **Approval route:** **`/join_requests`** (list) and **`/join_requests/:id`** (detail).
|
||||
- **Resend confirmation:** If not in Prio 1, create a separate ticket immediately.
|
||||
- **Pre-confirmation store:** DB only, same resource; no ETS, no stateless token. Token stored as **hash**; raw token only in the email link. **24h** retention for `pending_confirmation`; **hard-delete** of expired records via scheduled job (Oban cron) — see `lib/mix/tasks/join_requests.cleanup_expired.ex`.
|
||||
- **Confirmation route:** **`/confirm_join/:token`** so `starts_with?(path, "/confirm")` covers it.
|
||||
- **Public path for `/join`:** explicitly add `/join` to the plug's `public_path?/1` (e.g. in `CheckPagePermission`).
|
||||
- **JoinRequest schema:** status `pending_confirmation` | `submitted` | `approved` | `rejected`. Typed: **email** (required), **first_name**, **last_name** (optional). **form_data** (jsonb) + **schema_version** for the rest. **confirmation_token_hash**, **confirmation_token_expires_at**; **submitted_at**, **approved_at**, **rejected_at**, **reviewed_by_user_id**, **reviewed_by_display** (denormalized reviewer email for "Geprüft von" without loading User). Idempotent confirm (unique constraint on token hash, or update only when status is `pending_confirmation`).
|
||||
- **Approval outcome:** admin-configurable; default Member only (no User); optional "create User on approval" left for later.
|
||||
- **Rate limiting:** honeypot + rate limiting from the start (e.g. Hammer.Plug).
|
||||
- **Settings:** own section "Onboarding / Join"; `join_form_enabled` + field selection; display as list/badges; detailed UX in a separate subtask.
|
||||
- **Approval permission:** normal_user; JoinRequest read/update and the approval page added to normal_user; no new permission set.
|
||||
- **Approval route:** `/join_requests` (list), `/join_requests/:id` (detail).
|
||||
- **Resend confirmation:** if not in Prio 1, create a separate ticket immediately.
|
||||
|
||||
**Open for later:**
|
||||
|
||||
- Abuse metadata (IP hash etc.): classification and whether to store in Prio 1.
|
||||
- "Create User on approval" option: to be specified when implemented.
|
||||
- Invite link and OIDC JIT entry paths.
|
||||
**Open for later:** abuse metadata (IP hash etc.) classification and whether to store in Prio 1; "create User on approval" option (specify when implemented); invite link and OIDC JIT entry paths.
|
||||
|
||||
---
|
||||
|
||||
## 7. Definition of Done (Prio 1)
|
||||
|
||||
- Public `/join` page and confirmation route reachable without login; **`/join` explicitly** in public paths (plug and tests).
|
||||
- Flow: form submit → **JoinRequest created** in status `pending_confirmation` → confirmation email sent → user clicks link → **JoinRequest updated** to status `submitted`; no User or Member created by this flow.
|
||||
- Public `/join` page and confirmation route reachable without login; `/join` explicitly in public paths (plug + tests).
|
||||
- Flow: submit → JoinRequest `pending_confirmation` → email sent → click link → JoinRequest `submitted`; no User/Member created.
|
||||
- Anti-abuse: honeypot and rate limiting implemented and tested.
|
||||
- Cleanup: scheduled job hard-deletes JoinRequests in `pending_confirmation` older than 24h (or configured retention).
|
||||
- Page-permission and routing tests updated (including public-path coverage for `/join` and `/confirm_join/:token`).
|
||||
- Concept and decisions (§6) documented for use in implementation specs.
|
||||
- Cleanup: scheduled job hard-deletes `pending_confirmation` JoinRequests older than 24h.
|
||||
- Page-permission and routing tests updated (public-path coverage for `/join` and `/confirm_join/:token`).
|
||||
- Concept and decisions (§6) documented for implementation specs.
|
||||
|
||||
---
|
||||
|
||||
## 8. Implementation Plan (Subtasks)
|
||||
|
||||
**Resend confirmation** remains a separate ticket (see §2.5, §6).
|
||||
Resend confirmation remains a separate ticket (§2.5, §6).
|
||||
|
||||
### Prio 1 – Public Join (4 subtasks)
|
||||
**Prio 1 – Public Join (4 subtasks, all shipped):**
|
||||
|
||||
#### 1. JoinRequest resource and public policies **(done)**
|
||||
1. **JoinRequest resource and public policies** *(shipped)* — Ash resource per §2.3.2 (status, email required, first_name/last_name, form_data jsonb, schema_version, confirmation_token_hash + expiry, audit timestamps, source); migration; unique_index on confirmation_token_hash for idempotency. Public actions `submit` (create) and `confirm` (update) allowed with `actor: nil`; no system-actor fallback, no undocumented `authorize?: false`.
|
||||
2. **Submit and confirm flow** *(shipped)* — submit creates JoinRequest + sends confirmation email (reuse AshAuthentication sender); `/confirm_join/:token` verifies token (hash + lookup), updates to `submitted`, sets submitted_at, invalidates token (idempotent if already submitted); Oban hard-delete job for expired `pending_confirmation`.
|
||||
3. **Admin: Join form settings** *(shipped)* — "Onboarding / Join" settings section (§2.6): `join_form_enabled`, field selection (member_fields + custom fields), "required" per field; persisted; **server-side allowlist** available to subtask 4.
|
||||
4. **Public join page and anti-abuse** *(shipped)* — public `/join` route added to the plug's public path list; LiveView with fields from the allowlist; copy per §2.5; honeypot + rate limiting (Hammer.Plug); after-submit and expired-link copy; public-path tests updated to include `/join`.
|
||||
|
||||
- **Scope:** Ash resource `JoinRequest` per §2.3.2: status (`pending_confirmation`, `submitted`, `approved`, `rejected`), email (required), first_name, last_name (optional), form_data (jsonb), schema_version; confirmation_token_hash, confirmation_token_expires_at; submitted_at, approved_at, rejected_at, reviewed_by_user_id, source. Migration; unique_index on confirmation_token_hash (or equivalent for idempotency).
|
||||
- **Policies:** Public actions **submit** (create) and **confirm** (update) allowed with `actor: nil`; no system-actor fallback, no undocumented `authorize?: false`.
|
||||
- **Boundary:** No UI, no emails – only resource, persistence, and actions callable with nil actor.
|
||||
- **Done:** Resource and migration in place; tests for create/update with `actor: nil` and for idempotent confirm (same token twice → no second update).
|
||||
**Order and dependencies:** 1 → 2 (flow uses the resource); 3 before/parallel with 4 (form reads the allowlist from settings; MVP subtask 4 can use a default allowlist with 3 following shortly). Recommended: 1 → 2 → 3 → 4.
|
||||
|
||||
#### 2. Submit and confirm flow **(done)**
|
||||
**Step 2 – Approval (1 subtask, shipped):**
|
||||
|
||||
- **Scope:** Form submit → **create** JoinRequest (status `pending_confirmation`, token hash + expiry, form data) → send confirmation email (reuse AshAuthentication sender pattern). Route **`/confirm_join/:token`** → verify token (hash and lookup) → **update** JoinRequest to status `submitted`, set submitted_at, invalidate token (idempotent if already submitted). Optional: Oban (or similar) job to **hard-delete** JoinRequests in `pending_confirmation` with confirmation_token_expires_at older than 24h.
|
||||
- **Boundary:** No join-form UI, no admin settings – only backend create/update and email/route.
|
||||
- **Done:** Submit creates one JoinRequest; confirm updates it to submitted; double-click idempotent; expired token shows clear message; cleanup job implemented and documented. Tests for these cases.
|
||||
|
||||
#### 3. Admin: Join form settings **(done)**
|
||||
|
||||
- **Scope:** Section "Onboarding / Join" in global settings (§2.6): `join_form_enabled`, selection of join-form fields (from member_fields + custom fields), "required" per field. Persist (e.g. Setting or existing config). UI e.g. badges with remove + dropdown/modal to add (details in sub-subtask if needed).
|
||||
- **Boundary:** No public form – only save/load of config and **server-side allowlist** for use in subtask 4.
|
||||
- **Done:** Settings save/load; allowlist available in backend for join form; tests.
|
||||
|
||||
#### 4. Public join page and anti-abuse **(done)**
|
||||
|
||||
- **Scope:** Route **`/join`** (public). **Add `/join` to the page-permission plug’s public path list** so unauthenticated access is allowed. LiveView (or controller + form). Form fields from allowlist (subtask 3); copy per §2.5. **Honeypot** and **rate limiting** (e.g. Hammer.Plug) on join/submit. After submit: show "We have saved your details … click the link …". Expired-link page: clear message + "submit form again". Public-path tests updated to include `/join`.
|
||||
- **Boundary:** No approval UI, no User/Member creation – only public page, form, anti-abuse, and wiring to submit/confirm flow (subtask 2).
|
||||
- **Done:** Unauthenticated GET `/join` → 200; submit creates JoinRequest (pending_confirmation) and sends email; confirm updates to submitted; honeypot and rate limit tested; public-path tests updated.
|
||||
|
||||
### Order and dependencies
|
||||
|
||||
- **1 → 2:** Submit/confirm flow uses JoinRequest resource.
|
||||
- **3 before or in parallel with 4:** Form reads allowlist from settings; for MVP, subtask 4 can use a default allowlist and 3 can follow shortly after.
|
||||
- **Recommended order:** **1** → **2** → **3** → **4** (or 3 in parallel with 2 if two people work on it).
|
||||
|
||||
### Step 2 – Approval (1 subtask, later)
|
||||
|
||||
#### 5. Approval UI (Vorstand)
|
||||
|
||||
- **Route:** **`/join_requests`** (list), **`/join_requests/:id`** (detail). Full specification: §3.1.
|
||||
- **Scope:** List JoinRequests (status "submitted"), approve/reject actions; on approve create Member (no User in MVP). Permission: normal_user; add JoinRequest read/update (or approve/reject) and pages `/join_requests`, `/join_requests/:id` to PermissionSets. Populate audit fields (approved_at, rejected_at, reviewed_by_user_id). Promotion: JoinRequest → Member per §3.1 (mapping, defaults, idempotency).
|
||||
- **Boundary:** Separate ticket; builds on JoinRequest and existing Member creation.
|
||||
5. **Approval UI (Vorstand)** *(shipped)* — routes `/join_requests` (list) → `JoinRequestLive.Index`, `/join_requests/:id` (detail) → `JoinRequestLive.Show`; full spec in §3.1. Lists submitted JoinRequests, approve/reject; on approve creates a Member (no User in MVP). Permission: normal_user has JoinRequest read/update and the two pages in PermissionSets; audit fields populated; promotion JoinRequest → Member via `Mv.Membership.approve_join_request/2` per §3.1.
|
||||
|
||||
---
|
||||
|
||||
## 9. References
|
||||
|
||||
- `docs/roles-and-permissions-architecture.md` – Permission sets, roles, page permissions.
|
||||
- `docs/page-permission-route-coverage.md` – Public paths, plug behaviour, tests; add `/join_requests` and `/join_requests/:id` for Step 2 (normal_user).
|
||||
- `lib/mv_web/plugs/check_page_permission.ex` – Public path list; **add `/join`** in `public_path?/1`.
|
||||
- `lib/mv/accounts/user/senders/send_new_user_confirmation_email.ex` – Existing confirmation-email pattern (token, link, Mailer).
|
||||
- Hammer / Hammer.Plug (e.g. hexdocs.pm/hammer) – Rate limiting for Phoenix/Plug.
|
||||
- Issue #308 – Original feature/planning context.
|
||||
- `docs/roles-and-permissions-architecture.md` — permission sets, roles, page permissions.
|
||||
- `docs/page-permission-route-coverage.md` — public paths, plug behaviour, tests; covers `/join_requests` and `/join_requests/:id` for Step 2 (normal_user).
|
||||
- `lib/mv_web/plugs/check_page_permission.ex` — public path list; add `/join` in `public_path?/1`.
|
||||
- `lib/mv/authorization/checks/actor_is_nil.ex` — the actor:nil public-action check.
|
||||
- `lib/mix/tasks/join_requests.cleanup_expired.ex` — hard-delete of expired `pending_confirmation` JoinRequests (24h retention).
|
||||
- `lib/mv/accounts/user/senders/send_new_user_confirmation_email.ex` — existing confirmation-email pattern (token, link, Mailer).
|
||||
- Hammer / Hammer.Plug (hexdocs.pm/hammer) — rate limiting for Phoenix/Plug.
|
||||
- Issue #308 — original feature/planning context.
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue