docs(membership): condense membership, onboarding and import docs and align with the code
This commit is contained in:
parent
8d783276d0
commit
5d8f173529
4 changed files with 436 additions and 1904 deletions
|
|
@ -1,796 +1,172 @@
|
|||
# CSV Member Import v1 - Implementation Plan
|
||||
# CSV Member Import
|
||||
|
||||
**Version:** 1.0
|
||||
**Last Updated:** 2026-01-13
|
||||
**Status:** In Progress (Backend Complete, UI Complete, Tests Pending)
|
||||
**Related Documents:**
|
||||
- [Feature Roadmap](./feature-roadmap.md) - Overall feature planning
|
||||
Reference for how the CSV member import actually behaves. The end-to-end
|
||||
LiveView test (`test/mv_web/live/import_live_test.exs`) and future maintenance
|
||||
depend on the rules documented here.
|
||||
|
||||
## Implementation Status
|
||||
**Status:** implemented (backend + LiveView UI).
|
||||
|
||||
**Completed Issues:**
|
||||
- ✅ Issue #1: CSV Specification & Static Template Files
|
||||
- ✅ Issue #2: Import Service Module Skeleton
|
||||
- ✅ Issue #3: CSV Parsing + Delimiter Auto-Detection + BOM Handling
|
||||
- ✅ Issue #4: Header Normalization + Per-Header Mapping
|
||||
- ✅ Issue #5: Validation (Required Fields) + Error Formatting
|
||||
- ✅ Issue #6: Persistence via Ash Create + Per-Row Error Capture (with Error-Capping)
|
||||
- ✅ Issue #7: Admin Global Settings LiveView UI (Upload + Start Import + Results + Template Links)
|
||||
- ✅ Issue #8: Authorization + Limits
|
||||
- ✅ Issue #11: Custom Field Import (Backend + UI)
|
||||
Implementation:
|
||||
|
||||
**In Progress / Pending:**
|
||||
- ⏳ Issue #9: End-to-End LiveView Tests + Fixtures
|
||||
- ⏳ Issue #10: Documentation Polish
|
||||
- `lib/mv/membership/import/csv_parser.ex` — BOM stripping, delimiter detection, physical line numbering
|
||||
- `lib/mv/membership/import/header_mapper.ex` — header normalization + column mapping
|
||||
- `lib/mv/membership/import/column_resolver.ex` — read-only resolution of groups + fee-type columns (preview)
|
||||
- `lib/mv/membership/import/member_csv.ex` — `prepare/2`, `process_chunk/4`, validation, member creation
|
||||
- `lib/mv/membership/import/import_runner.ex` — orchestration glue
|
||||
- `lib/mv_web/live/import_live.ex` (+ `import_live/components.ex`) — UI, state machine, chunk driving
|
||||
- `lib/mv_web/controllers/import_template_controller.ex` — on-the-fly template generation
|
||||
|
||||
**Latest Update:** CSV Import UI fully implemented in GlobalSettingsLive with chunk processing, progress tracking, error display, and custom field support (2026-01-13)
|
||||
## Scope
|
||||
|
||||
---
|
||||
Admin-only bulk creation of members from an uploaded CSV.
|
||||
|
||||
## Table of Contents
|
||||
- **Create only** — no upsert/update of existing members.
|
||||
- **No deduplication** — a duplicate email fails its row (unique constraint) and is reported as an error.
|
||||
- **Best-effort, row-by-row** — no transactional rollback; a failed row does not abort the import.
|
||||
- **No background jobs** — progress is driven via LiveView `handle_info` chunk messages.
|
||||
- **Errors shown in UI only** — no error-CSV export.
|
||||
|
||||
- [Overview & Scope](#overview--scope)
|
||||
- [UX Flow](#ux-flow)
|
||||
- [CSV Specification](#csv-specification)
|
||||
- [Technical Design Notes](#technical-design-notes)
|
||||
- [Implementation Issues](#implementation-issues)
|
||||
- [Rollout & Risks](#rollout--risks)
|
||||
Out of scope: upsert, mapping wizard, transactional all-or-nothing, error export, import history/audit.
|
||||
|
||||
---
|
||||
## UI Flow
|
||||
|
||||
## Overview & Scope
|
||||
- **Route:** `/admin/import` (LiveView `MvWeb.ImportLive`). Template downloads:
|
||||
`/admin/import/template/en` and `/admin/import/template/de` (dynamic controller, not static files).
|
||||
- **Authorization:** requires `can?(:create, Mv.Membership.Member)`. Non-admins are
|
||||
redirected with a "don't have permission" flash. The import section, the template
|
||||
controller, and the `start_import` event all enforce this.
|
||||
- **Upload:** `allow_upload(:csv_file, accept: .csv, max_entries: 1, auto_upload: true)`.
|
||||
File size limit enforced by `max_file_size`.
|
||||
- **State machine** (`@import_status`): `idle → preview → running → done|error`.
|
||||
- **start_import** parses + resolves the file and transitions to **preview**. This step
|
||||
is **read-only**: no members are created yet. The preview shows the column mapping,
|
||||
sample rows, groups that exist vs. would be created, and fee-type/unknown-column warnings.
|
||||
- **confirm_import** begins processing and creates members chunk by chunk.
|
||||
- **Results:** success count, failure count, error list (each with CSV line number, message,
|
||||
optional field), warnings, and a truncation notice when errors exceed the cap.
|
||||
|
||||
### What We're Building
|
||||
## Limits
|
||||
|
||||
A **basic CSV member import feature** that allows administrators to upload a CSV file and import new members into the system. This is a **v1 minimal implementation** focused on establishing the import structure without advanced features.
|
||||
- **Max file size:** configurable via `config :mv, csv_import: [max_file_size_mb: ...]` (enforced by `allow_upload`).
|
||||
- **Max rows:** configurable via `config :mv, csv_import: [max_rows: ...]`, default **1000**, excluding header. Enforced in `MemberCSV.prepare/2`; exceeding it yields an error containing `"exceeds"`.
|
||||
- **Chunk size:** 200 rows per chunk.
|
||||
- **Error cap:** 50 errors collected per import overall (`failed` count stays accurate; `errors_truncated?` flag set when exceeded).
|
||||
|
||||
**Core Functionality (v1 Minimal):**
|
||||
- Upload CSV file via LiveView file upload
|
||||
- Parse CSV with bilingual header support for core member fields (English/German)
|
||||
- Auto-detect delimiter (`;` or `,`) using header recognition
|
||||
- Map CSV columns to core member fields (`first_name`, `last_name`, `email`, `street`, `postal_code`, `city`, `country`)
|
||||
- **Import custom field values** - Map CSV columns to existing custom fields by name (unknown custom field columns will be ignored with a warning)
|
||||
- Validate each row (required field: `email`)
|
||||
- Create members via Ash resource (one-by-one, **no background jobs**, processed in chunks of 200 rows via LiveView messages)
|
||||
- Display import results: success count, error count, and error details
|
||||
- Provide static CSV templates (EN/DE)
|
||||
## Parsing (`CsvParser.parse/1`)
|
||||
|
||||
**Key Constraints (v1):**
|
||||
- ✅ **Admin-only feature**
|
||||
- ✅ **No upsert** (create only)
|
||||
- ✅ **No deduplication** (duplicate emails fail and show as errors)
|
||||
- ✅ **No mapping wizard** (fixed header mapping via bilingual variants)
|
||||
- ✅ **No background jobs** (progress via LiveView `handle_info`)
|
||||
- ✅ **Best-effort import** (row-by-row, no rollback)
|
||||
- ✅ **UI-only error display** (no error CSV export)
|
||||
- ✅ **Safety limits** (10 MB, 1,000 rows, chunks of 200)
|
||||
- Content must be **valid UTF-8** (else error). Empty content / empty header row are errors.
|
||||
- **UTF-8 BOM is stripped first**, before any header handling.
|
||||
- Line endings normalized: `\r\n`, `\r`, `\n` all handled.
|
||||
- **Delimiter auto-detection:** parse the header with both `;` and `,` parsers (NimbleCSV,
|
||||
quote-aware), count non-empty fields each yields, pick the higher; **`;` wins ties**;
|
||||
default `;`.
|
||||
- **Quoting:** double-quote quoting; `""` inside a quoted field is a literal `"`. Newlines
|
||||
inside quoted fields are supported — the record keeps its **start** line number.
|
||||
- **Physical line numbers:** rows are returned as `{csv_line_number, values}` where the line
|
||||
number is the physical 1-based line in the file (header is line 1, first data row is line 2).
|
||||
**Empty lines are skipped but do not shift numbering** — downstream code must use the
|
||||
parser's line numbers, never recompute from row index. (Test asserts an invalid row after a
|
||||
skipped empty line still reports its true physical line, e.g. `Line 4`.)
|
||||
- Completely empty rows are skipped. An unparsable row produces an error naming its line number.
|
||||
|
||||
### Out of Scope (v1)
|
||||
## Header Mapping & Normalization (`HeaderMapper`)
|
||||
|
||||
**Deferred to Future Versions:**
|
||||
- ❌ Upsert/update existing members
|
||||
- ❌ Advanced deduplication strategies
|
||||
- ❌ Column mapping wizard UI
|
||||
- ❌ Background job processing (Oban/GenStage)
|
||||
- ❌ Transactional all-or-nothing import
|
||||
- ❌ Error CSV export/download
|
||||
- ❌ Batch validation preview before import
|
||||
- ❌ Dynamic template generation
|
||||
- ❌ Import history/audit log
|
||||
- ❌ Import templates for other entities
|
||||
**`normalize_header/1`** (applied identically to incoming headers, mapping variants, custom
|
||||
field names, group names, and fee-type names):
|
||||
|
||||
---
|
||||
1. trim, lowercase
|
||||
2. transliterate German chars: `ß → ss`, `ä → ae`, `ö → oe`, `ü → ue` (and uppercase forms)
|
||||
3. unify hyphen variants (en dash U+2013, em dash U+2014, minus U+2212 → `-`)
|
||||
4. punctuation to spaces: `_`, `()[]{}`, `/`, `\` → space
|
||||
5. **remove all whitespace** (so `first name` == `firstname`)
|
||||
6. final trim
|
||||
|
||||
## UX Flow
|
||||
Matching is on the fully normalized string.
|
||||
|
||||
### Access & Location
|
||||
**Required field:** `email`. Missing it aborts `prepare` with a "Missing required header" error.
|
||||
|
||||
**Entry Point:**
|
||||
- **Location:** Global Settings page (`/settings`)
|
||||
- **UI Element:** New section "Import Members (CSV)" below "Custom Fields" section
|
||||
- **Access Control:** Admin-only (enforced at LiveView event level, not entire `/settings` route)
|
||||
**Unknown member-field columns:** ignored (no error). If an unknown column looks like it
|
||||
could be a custom field that does not exist, a **warning** is emitted (import continues).
|
||||
|
||||
### User Journey
|
||||
**Duplicate headers** mapping to the same canonical field (or same custom field) are an error.
|
||||
|
||||
1. **Navigate to Global Settings**
|
||||
2. **Access Import Section**
|
||||
- **Important notice:** Custom fields should be created in Mila before importing CSV files with custom field columns (unknown columns will be ignored with a warning)
|
||||
- Upload area (drag & drop or file picker)
|
||||
- Template download links (English / German)
|
||||
- Help text explaining CSV format and custom field requirements
|
||||
3. **Ensure Custom Fields Exist (if importing custom fields)**
|
||||
- Navigate to Custom Fields section and create required custom fields
|
||||
- Note the name/identifier for each custom field (used as CSV header)
|
||||
4. **Download Template (Optional)**
|
||||
5. **Prepare CSV File**
|
||||
- Include custom field columns using the custom field name as header (e.g., `membership_number`, `birth_date`)
|
||||
6. **Upload CSV**
|
||||
7. **Start Import**
|
||||
- Runs server-side via LiveView messages (may take up to ~30 seconds for large files)
|
||||
- Warning messages if custom field columns reference non-existent custom fields (columns will be ignored)
|
||||
8. **View Results**
|
||||
- Success count
|
||||
- Error count
|
||||
- First 50 errors, each with:
|
||||
- **CSV line number** (header is line 1, first data record begins at line 2)
|
||||
- Error message
|
||||
- Field name (if applicable)
|
||||
### Supported member fields and header variants
|
||||
|
||||
### Error Handling
|
||||
Source of truth is `@member_field_variants_raw` in `header_mapper.ex`. Variants below are
|
||||
illustrative; matching is via normalization, so casing/hyphen/whitespace differences all collapse.
|
||||
|
||||
- **File too large:** Flash error before upload starts
|
||||
- **Too many rows:** Flash error before import starts
|
||||
- **Invalid CSV format:** Error shown in results
|
||||
- **Partial success:** Results show both success and error counts
|
||||
|
||||
---
|
||||
|
||||
## CSV Specification
|
||||
|
||||
### Delimiter
|
||||
|
||||
**Recommended:** Semicolon (`;`)
|
||||
**Supported:** `;` and `,`
|
||||
|
||||
**Auto-Detection (Header Recognition):**
|
||||
- Remove UTF-8 BOM *first*
|
||||
- Extract header record and try parsing with both delimiters
|
||||
- For each delimiter, count how many recognized headers are present (via normalized variants)
|
||||
- Choose delimiter with higher recognition; prefer `;` if tied
|
||||
- If neither yields recognized headers, default to `;`
|
||||
|
||||
### Quoting Rules
|
||||
|
||||
- Fields may be quoted with double quotes (`"`)
|
||||
- Escaped quotes: `""` inside quoted field represents a single `"`
|
||||
- **v1 assumption:** CSV records do **not** contain embedded newlines inside quoted fields. (If they do, parsing may fail or line numbers may be inaccurate.)
|
||||
|
||||
### Column Headers
|
||||
|
||||
**v1 Supported Fields:**
|
||||
|
||||
**Core Member Fields (all importable):**
|
||||
- `email` / `E-Mail` (required)
|
||||
- `first_name` / `Vorname` (optional)
|
||||
- `last_name` / `Nachname` (optional)
|
||||
- `join_date` / `Beitrittsdatum` (optional, ISO-8601 date)
|
||||
- `exit_date` / `Austrittsdatum` (optional, ISO-8601 date)
|
||||
- `notes` / `Notizen` (optional)
|
||||
- `country` / `Land` / `Staat` (optional)
|
||||
- `city` / `Stadt` (optional)
|
||||
- `street` / `Straße` (optional)
|
||||
- `house_number` / `Hausnummer` / `Nr.` (optional)
|
||||
- `postal_code` / `PLZ` / `Postleitzahl` (optional)
|
||||
- `membership_fee_start_date` / `Beitragsbeginn` (optional, ISO-8601 date)
|
||||
|
||||
Address column order in import/export matches the members overview: country, city, street, house number, postal code.
|
||||
|
||||
**Not supported for import (by design):**
|
||||
- **membership_fee_status** – Computed field (from fee cycles). Not stored; export-only.
|
||||
- **groups** – Many-to-many relationship. Would require resolving group names to IDs; not in current scope.
|
||||
- **membership_fee_type_id** – Foreign key; could be added later (e.g. resolve type name to ID).
|
||||
|
||||
**Custom Fields:**
|
||||
- Any custom field column using the custom field's **name** as the header (e.g., `membership_number`, `birth_date`)
|
||||
- **Important:** Custom fields must be created in Mila before importing. The CSV header must match the custom field name exactly (same normalization as member fields).
|
||||
- **Behavior:** If the CSV contains custom field columns that don't exist in Mila, a warning message will be shown and those columns will be ignored during import.
|
||||
- **Value Validation:** Custom field values are validated according to the custom field type:
|
||||
- **string**: Any text value (trimmed)
|
||||
- **integer**: Must be a valid integer (e.g., `42`, `-10`). Invalid values will cause a row error with the custom field name and reason.
|
||||
- **boolean**: Accepts `true`, `false`, `1`, `0`, `yes`, `no`, `ja`, `nein` (case-insensitive). Invalid values will cause a row error.
|
||||
- **date**: Must be in ISO-8601 format (YYYY-MM-DD, e.g., `2024-01-15`). Invalid values will cause a row error.
|
||||
- **email**: Must be a valid email format (contains `@`, 5-254 characters, valid format). Invalid values will cause a row error.
|
||||
- **Error Messages:** Custom field validation errors are included in the import error list with format: `custom_field: <name> – <reason>` (e.g., `custom_field: Alter – expected integer, got: abc`)
|
||||
|
||||
**Member Field Header Mapping:**
|
||||
|
||||
| Canonical Field | English Variants | German Variants |
|
||||
| Canonical | Example accepted headers (EN / DE) | Notes |
|
||||
|---|---|---|
|
||||
| `first_name` | `first_name`, `firstname` | `Vorname`, `vorname` |
|
||||
| `last_name` | `last_name`, `lastname`, `surname` | `Nachname`, `nachname`, `Familienname` |
|
||||
| `email` | `email`, `e-mail`, `e_mail` | `E-Mail`, `e-mail`, `e_mail` |
|
||||
| `join_date` | `join date`, `join_date` | `Beitrittsdatum`, `beitritts-datum` |
|
||||
| `exit_date` | `exit date`, `exit_date` | `Austrittsdatum`, `austritts-datum` |
|
||||
| `notes` | `notes` | `Notizen`, `bemerkungen` |
|
||||
| `street` | `street`, `address` | `Straße`, `strasse`, `Strasse` |
|
||||
| `house_number` | `house number`, `house_number`, `house no` | `Hausnummer`, `Nr`, `Nr.`, `Nummer` |
|
||||
| `postal_code` | `postal_code`, `zip`, `postcode` | `PLZ`, `plz`, `Postleitzahl`, `postleitzahl` |
|
||||
| `city` | `city`, `town` | `Stadt`, `stadt`, `Ort` |
|
||||
| `country` | `country` | `Land`, `land`, `Staat`, `staat` |
|
||||
| `membership_fee_start_date` | `membership fee start date`, `membership_fee_start_date`, `fee start` | `Beitragsbeginn`, `beitrags-beginn` |
|
||||
|
||||
**Header Normalization (used consistently for both input headers AND mapping variants):**
|
||||
- Trim whitespace
|
||||
- Convert to lowercase
|
||||
- Normalize Unicode: `ß` → `ss` (e.g., `Straße` → `strasse`)
|
||||
- Replace hyphens/whitespace with underscores: `E-Mail` → `e_mail`, `phone number` → `phone_number`
|
||||
- Collapse multiple underscores: `e__mail` → `e_mail`
|
||||
- Case-insensitive matching
|
||||
|
||||
**Unknown columns:** ignored (no error)
|
||||
|
||||
**Required fields:** `email`
|
||||
|
||||
**Custom Field Columns:**
|
||||
- Custom field columns are identified by matching the normalized CSV header to the custom field `name` (not slug)
|
||||
- Same normalization rules apply as for member fields (trim, lowercase, Unicode normalization, underscore replacement)
|
||||
- Unknown custom field columns (non-existent names) will be ignored with a warning message
|
||||
|
||||
### CSV Template Files
|
||||
|
||||
**Location:**
|
||||
- `priv/static/templates/member_import_en.csv`
|
||||
- `priv/static/templates/member_import_de.csv`
|
||||
|
||||
**Content:**
|
||||
- Header row with required + common optional fields
|
||||
- **Note:** Custom field columns are not included in templates by default (users add them based on their custom field configuration)
|
||||
- One example row
|
||||
- Uses semicolon delimiter (`;`)
|
||||
- UTF-8 encoding **with BOM** (Excel compatibility)
|
||||
|
||||
**Template Access:**
|
||||
- Templates are static files in `priv/static/templates/`
|
||||
- Served at:
|
||||
- `/templates/member_import_en.csv`
|
||||
- `/templates/member_import_de.csv`
|
||||
- In LiveView, link them using Phoenix static path helpers (e.g. `~p` or `Routes.static_path/2`, depending on Phoenix version).
|
||||
|
||||
**Example Usage in LiveView Templates:**
|
||||
|
||||
```heex
|
||||
<!-- Using ~p sigil (Phoenix 1.7+) -->
|
||||
<.link href={~p"/templates/member_import_en.csv"} download>
|
||||
<%= gettext("Download English Template") %>
|
||||
</.link>
|
||||
|
||||
<.link href={~p"/templates/member_import_de.csv"} download>
|
||||
<%= gettext("Download German Template") %>
|
||||
</.link>
|
||||
|
||||
<!-- Alternative: Using Routes.static_path/2 -->
|
||||
<.link href={Routes.static_path(MvWeb.Endpoint, "/templates/member_import_en.csv")} download>
|
||||
<%= gettext("Download English Template") %>
|
||||
</.link>
|
||||
```
|
||||
|
||||
**Note:** The `templates` directory must be included in `MvWeb.static_paths()` (configured in `lib/mv_web.ex`) for the files to be served.
|
||||
|
||||
### File Limits
|
||||
|
||||
- **Max file size:** 10 MB
|
||||
- **Max rows:** 1,000 rows (excluding header)
|
||||
- **Processing:** chunks of 200 (via LiveView messages)
|
||||
- **Encoding:** UTF-8 (BOM handled)
|
||||
|
||||
---
|
||||
|
||||
## Technical Design Notes
|
||||
|
||||
### Architecture Overview
|
||||
|
||||
```
|
||||
┌─────────────────┐
|
||||
│ LiveView UI │ (GlobalSettingsLive or component)
|
||||
│ - Upload area │
|
||||
│ - Progress │
|
||||
│ - Results │
|
||||
└────────┬────────┘
|
||||
│ prepare
|
||||
▼
|
||||
┌─────────────────────────────┐
|
||||
│ Import Service │ (Mv.Membership.Import.MemberCSV)
|
||||
│ - parse + map + limit checks│ -> returns import_state
|
||||
│ - process_chunk(chunk) │ -> returns chunk results
|
||||
└────────┬────────────────────┘
|
||||
│ create
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ Ash Resource │ (Mv.Membership.Member)
|
||||
│ - Create │
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
### Technology Stack
|
||||
|
||||
- **Phoenix LiveView:** file upload via `allow_upload/3`
|
||||
- **NimbleCSV:** CSV parsing (add explicit dependency if missing)
|
||||
- **Ash Resource:** member creation via `Membership.create_member/1`
|
||||
- **Gettext:** bilingual UI/error messages
|
||||
|
||||
### Module Structure
|
||||
|
||||
**New Modules:**
|
||||
- `lib/mv/membership/import/member_csv.ex` - import orchestration + chunk processing + custom field handling
|
||||
- `lib/mv/membership/import/csv_parser.ex` - delimiter detection + parsing + BOM handling
|
||||
- `lib/mv/membership/import/header_mapper.ex` - normalization + header mapping (core fields + custom fields)
|
||||
|
||||
**Modified Modules:**
|
||||
- `lib/mv_web/live/global_settings_live.ex` - render import section, handle upload/events/messages
|
||||
|
||||
### Data Flow
|
||||
|
||||
1. **Upload:** LiveView receives file via `allow_upload`
|
||||
2. **Consume:** `consume_uploaded_entries/3` reads file content
|
||||
3. **Prepare:** `MemberCSV.prepare/2`
|
||||
- Strip BOM
|
||||
- Detect delimiter (header recognition)
|
||||
- Parse header + rows
|
||||
- Map headers to canonical fields (core member fields)
|
||||
- **Query existing custom fields and map custom field columns by name** (using same normalization as member fields)
|
||||
- **Warn about unknown custom field columns** (non-existent names will be ignored with warning)
|
||||
- Early abort if required headers missing
|
||||
- Row count check
|
||||
- Return `import_state` containing chunks, column_map, and custom_field_map
|
||||
4. **Process:** LiveView drives chunk processing via `handle_info`
|
||||
- For each chunk: validate + create member + create custom field values + collect errors
|
||||
5. **Results:** LiveView shows progress + final summary
|
||||
|
||||
### Types & Key Consistency
|
||||
|
||||
- **Raw CSV parsing:** returns headers as list of strings, and rows **with csv line numbers**
|
||||
- **Header mapping:** operates on normalized strings; mapping table variants are normalized once
|
||||
- **Ash attrs:** built as atom-keyed map (`%{first_name: ..., ...}`)
|
||||
|
||||
### Error Model
|
||||
|
||||
```elixir
|
||||
%{
|
||||
csv_line_number: 5, # physical line number in the CSV file
|
||||
field: :email, # optional
|
||||
message: "is not a valid email"
|
||||
}
|
||||
```
|
||||
|
||||
### CSV Line Numbers (Important)
|
||||
|
||||
To keep error reporting user-friendly and accurate, **row errors must reference the physical line number in the original file**, even if empty lines are skipped.
|
||||
|
||||
**Design decision:** the parser returns rows as:
|
||||
|
||||
```elixir
|
||||
rows :: [{csv_line_number :: pos_integer(), row_map :: map()}]
|
||||
```
|
||||
|
||||
Downstream logic must **not** recompute line numbers from row indexes.
|
||||
|
||||
### Authorization
|
||||
|
||||
**Enforcement points:**
|
||||
1. **LiveView event level:** check admin permission in `handle_event("start_import", ...)`
|
||||
2. **UI level:** render import section only for admin users
|
||||
3. **Static templates:** public assets (no authorization needed)
|
||||
|
||||
Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string checks where possible.
|
||||
|
||||
### Safety Limits
|
||||
|
||||
- File size enforced by `allow_upload` (`max_file_size`)
|
||||
- Row count enforced in `MemberCSV.prepare/2` before processing starts
|
||||
- Chunking is done via **LiveView `handle_info` loop** (sequential, cooperative scheduling)
|
||||
|
||||
---
|
||||
|
||||
## Implementation Issues
|
||||
|
||||
### Issue #1: CSV Specification & Static Template Files
|
||||
|
||||
**Dependencies:** None
|
||||
|
||||
**Status:** ✅ **COMPLETED**
|
||||
|
||||
**Goal:** Define CSV contract and add static templates.
|
||||
|
||||
**Tasks:**
|
||||
- [x] Finalize header mapping variants
|
||||
- [x] Document normalization rules
|
||||
- [x] Document delimiter detection strategy
|
||||
- [x] Create templates in `priv/static/templates/` (UTF-8 with BOM)
|
||||
- `member_import_en.csv` with English headers
|
||||
- `member_import_de.csv` with German headers
|
||||
- [x] Document template URLs and how to link them from LiveView
|
||||
- [x] Document line number semantics (physical CSV line numbers)
|
||||
- [x] Templates included in `MvWeb.static_paths()` configuration
|
||||
|
||||
**Definition of Done:**
|
||||
- [x] Templates open cleanly in Excel/LibreOffice
|
||||
- [x] CSV spec section complete
|
||||
|
||||
---
|
||||
|
||||
### Issue #2: Import Service Module Skeleton
|
||||
|
||||
**Dependencies:** None
|
||||
|
||||
**Status:** ✅ **COMPLETED**
|
||||
|
||||
**Goal:** Create service API and error types.
|
||||
|
||||
**API (recommended):**
|
||||
- `prepare/2` — parse + map + limit checks, returns import_state
|
||||
- `process_chunk/4` — process one chunk (pure-ish), returns per-chunk results
|
||||
|
||||
**Tasks:**
|
||||
- [x] Create `lib/mv/membership/import/member_csv.ex`
|
||||
- [x] Define public function: `prepare/2 (file_content, opts \\ [])`
|
||||
- [x] Define public function: `process_chunk/4 (chunk_rows_with_lines, column_map, custom_field_map, opts \\ [])`
|
||||
- [x] Define error struct: `%MemberCSV.Error{csv_line_number: integer, field: atom | nil, message: String.t}`
|
||||
- [x] Document module + API
|
||||
|
||||
---
|
||||
|
||||
### Issue #3: CSV Parsing + Delimiter Auto-Detection + BOM Handling
|
||||
|
||||
**Dependencies:** Issue #2
|
||||
|
||||
**Status:** ✅ **COMPLETED**
|
||||
|
||||
**Goal:** Parse CSV robustly with correct delimiter detection and BOM handling.
|
||||
|
||||
**Tasks:**
|
||||
- [x] Verify/add NimbleCSV dependency (`{:nimble_csv, "~> 1.0"}`)
|
||||
- [x] Create `lib/mv/membership/import/csv_parser.ex`
|
||||
- [x] Implement `strip_bom/1` and apply it **before** any header handling
|
||||
- [x] Handle `\r\n` and `\n` line endings (trim `\r` on header record)
|
||||
- [x] Detect delimiter via header recognition (try `;` and `,`)
|
||||
- [x] Parse CSV and return:
|
||||
- `headers :: [String.t()]`
|
||||
- `rows :: [{csv_line_number, [String.t()]}]` with correct physical line numbers
|
||||
- [x] Skip completely empty records (but preserve correct physical line numbers)
|
||||
- [x] Return `{:ok, headers, rows}` or `{:error, reason}`
|
||||
|
||||
**Definition of Done:**
|
||||
- [x] BOM handling works (Excel exports)
|
||||
- [x] Delimiter detection works reliably
|
||||
- [x] Rows carry correct `csv_line_number`
|
||||
|
||||
---
|
||||
|
||||
### Issue #4: Header Normalization + Per-Header Mapping (No Language Detection)
|
||||
|
||||
**Dependencies:** Issue #3
|
||||
|
||||
**Status:** ✅ **COMPLETED**
|
||||
|
||||
**Goal:** Map each header individually to canonical fields (normalized comparison).
|
||||
|
||||
**Tasks:**
|
||||
- [x] Create `lib/mv/membership/import/header_mapper.ex`
|
||||
- [x] Implement `normalize_header/1`
|
||||
- [x] Normalize mapping variants once and compare normalized strings
|
||||
- [x] Build `column_map` (canonical field -> column index)
|
||||
- [x] **Early abort if required headers missing** (`email`)
|
||||
- [x] Ignore unknown columns (member fields only)
|
||||
- [x] **Separate custom field column detection** (by name, with normalization)
|
||||
|
||||
**Definition of Done:**
|
||||
- [x] English/German headers map correctly
|
||||
- [x] Missing required columns fails fast
|
||||
|
||||
---
|
||||
|
||||
### Issue #5: Validation (Required Fields) + Error Formatting
|
||||
|
||||
**Dependencies:** Issue #4
|
||||
|
||||
**Status:** ✅ **COMPLETED**
|
||||
|
||||
**Goal:** Validate each row and return structured, translatable errors.
|
||||
|
||||
**Tasks:**
|
||||
- [x] Implement `validate_row/3 (row_map, csv_line_number, opts)`
|
||||
- [x] Required field presence (`email`)
|
||||
- [x] Email format validation (EctoCommons.EmailValidator)
|
||||
- [x] Trim values before validation
|
||||
- [x] Gettext-backed error messages
|
||||
|
||||
---
|
||||
|
||||
### Issue #6: Persistence via Ash Create + Per-Row Error Capture (Chunked Processing)
|
||||
|
||||
**Dependencies:** Issue #5
|
||||
|
||||
**Status:** ✅ **COMPLETED**
|
||||
|
||||
**Goal:** Create members and capture errors per row with correct CSV line numbers.
|
||||
|
||||
**Tasks:**
|
||||
- [x] Implement `process_chunk/4` in service:
|
||||
- Input: `[{csv_line_number, row_map}]`
|
||||
- Validate + create sequentially
|
||||
- Collect counts + first 50 errors (per import overall; LiveView enforces cap across chunks)
|
||||
- **Error-Capping:** Supports `existing_error_count` and `max_errors` in opts (default: 50)
|
||||
- **Error-Capping:** Only collects errors if under limit, but continues processing all rows
|
||||
- **Error-Capping:** `failed` count is always accurate, even when errors are capped
|
||||
- [x] Implement Ash error formatter helper:
|
||||
- Convert `Ash.Error.Invalid` into `%MemberCSV.Error{}`
|
||||
- Prefer field-level errors where possible (attach `field` atom)
|
||||
- Handle unique email constraint error as user-friendly message
|
||||
- [x] Map row_map to Ash attrs (`%{first_name: ..., ...}`)
|
||||
- [x] Custom field value processing and creation
|
||||
|
||||
**Important:** **Do not recompute line numbers** in this layer—use the ones provided by the parser.
|
||||
|
||||
**Implementation Notes:**
|
||||
- `process_chunk/4` accepts `opts` with `existing_error_count` and `max_errors` for error capping across chunks
|
||||
- Error capping respects the limit per import overall (not per chunk)
|
||||
- Processing continues even after error limit is reached (for accurate counts)
|
||||
|
||||
---
|
||||
|
||||
### Issue #7: Admin Global Settings LiveView UI (Upload + Start Import + Results + Template Links)
|
||||
|
||||
**Dependencies:** Issue #6
|
||||
|
||||
**Status:** ✅ **COMPLETED**
|
||||
|
||||
**Goal:** UI section with upload, progress, results, and template links.
|
||||
|
||||
**Tasks:**
|
||||
- [x] Render import section only for admins
|
||||
- [x] **Add prominent UI notice about custom fields:**
|
||||
- Display alert/info box: "Custom fields must be created in Mila before importing CSV files with custom field columns"
|
||||
- Explain: "Use the custom field name as the CSV column header (same normalization as member fields applies)"
|
||||
- Add link to custom fields management section
|
||||
- [x] Configure `allow_upload/3`:
|
||||
- `.csv` only, `max_entries: 1`, `max_file_size: 10MB`, `auto_upload: true` (auto-upload enabled for better UX)
|
||||
- [x] `handle_event("start_import", ...)`:
|
||||
- Admin permission check
|
||||
- Consume upload -> read file content
|
||||
- Call `MemberCSV.prepare/2`
|
||||
- Store `import_state` in assigns (chunks + column_map + metadata)
|
||||
- Initialize progress assigns
|
||||
- `send(self(), {:process_chunk, 0})`
|
||||
- [x] `handle_info({:process_chunk, idx}, socket)`:
|
||||
- Fetch chunk from `import_state`
|
||||
- Call `MemberCSV.process_chunk/4` with error capping support
|
||||
- Merge counts/errors into progress assigns (cap errors at 50 overall)
|
||||
- Schedule next chunk (or finish and show results)
|
||||
- Async task processing with SQL sandbox support for tests
|
||||
- [x] Results UI:
|
||||
- Success count
|
||||
- Failure count
|
||||
- Error list (line number + message + field)
|
||||
- **Warning messages for unknown custom field columns** (non-existent names) shown in results
|
||||
- Progress indicator during import
|
||||
- Error truncation notice when errors exceed limit
|
||||
|
||||
**Template links:**
|
||||
- [x] Link `/templates/member_import_en.csv` and `/templates/member_import_de.csv` via Phoenix static path helpers.
|
||||
|
||||
**Definition of Done:**
|
||||
- [x] Upload area with drag & drop support
|
||||
- [x] Template download links (EN/DE)
|
||||
- [x] Progress tracking during import
|
||||
- [x] Results display with success/error counts
|
||||
- [x] Error list with line numbers and field information
|
||||
- [x] Warning display for unknown custom field columns
|
||||
- [x] Admin-only access control
|
||||
- [x] Async chunk processing with proper error handling
|
||||
|
||||
---
|
||||
|
||||
### Issue #8: Authorization + Limits
|
||||
|
||||
**Dependencies:** None (can be parallelized)
|
||||
|
||||
**Status:** ✅ **COMPLETED**
|
||||
|
||||
**Goal:** Ensure admin-only access and enforce limits.
|
||||
|
||||
**Tasks:**
|
||||
- [x] Admin check in start import event handler (via `Authorization.can?/3`)
|
||||
- [x] File size enforced in upload config (`max_file_size: 10MB`)
|
||||
- [x] Row limit enforced in `MemberCSV.prepare/2` (max_rows: 1000, configurable via opts)
|
||||
- [x] Chunk size limit (200 rows per chunk)
|
||||
- [x] Error limit (50 errors per import)
|
||||
- [x] UI-level authorization check (import section only visible to admins)
|
||||
- [x] Event-level authorization check (prevents unauthorized import attempts)
|
||||
|
||||
**Implementation Notes:**
|
||||
- File size limit: 10 MB (10,485,760 bytes) enforced via `allow_upload/3`
|
||||
- Row limit: 1,000 rows (excluding header) enforced in `MemberCSV.prepare/2`
|
||||
- Chunk size: 200 rows per chunk (configurable via opts)
|
||||
- Error limit: 50 errors per import (configurable via `@max_errors`)
|
||||
- Authorization uses `MvWeb.Authorization.can?/3` with `:create` permission on `Mv.Membership.Member`
|
||||
|
||||
**Definition of Done:**
|
||||
- [x] Admin-only access enforced at UI and event level
|
||||
- [x] File size limit enforced
|
||||
- [x] Row count limit enforced
|
||||
- [x] Chunk processing with size limits
|
||||
- [x] Error capping implemented
|
||||
|
||||
---
|
||||
|
||||
### Issue #9: End-to-End LiveView Tests + Fixtures
|
||||
|
||||
**Dependencies:** Issue #7 and #8
|
||||
|
||||
**Tasks:**
|
||||
- [ ] Fixtures:
|
||||
- valid EN/DE (core fields only)
|
||||
- valid with custom fields
|
||||
- invalid
|
||||
- unknown custom field name (non-existent, should show warning)
|
||||
- too many rows (1,001)
|
||||
- BOM + `;` delimiter fixture
|
||||
- fixture with empty line(s) to validate correct line numbers
|
||||
- [ ] LiveView tests:
|
||||
- admin sees section, non-admin does not
|
||||
- upload + start import
|
||||
- success + error rendering
|
||||
- row limit + file size errors
|
||||
- custom field import success
|
||||
- custom field import warning (non-existent name, column ignored)
|
||||
|
||||
---
|
||||
|
||||
### Issue #10: Documentation Polish (Inline Help Text + Docs)
|
||||
|
||||
**Dependencies:** Issue #9
|
||||
|
||||
**Tasks:**
|
||||
- [ ] UI help text + translations
|
||||
- [ ] CHANGELOG entry
|
||||
- [ ] Ensure moduledocs/docs
|
||||
|
||||
---
|
||||
|
||||
### Issue #11: Custom Field Import
|
||||
|
||||
**Dependencies:** Issue #6 (Persistence)
|
||||
|
||||
**Priority:** High (Core v1 Feature)
|
||||
|
||||
**Status:** ✅ **COMPLETED** (Backend + UI Implementation)
|
||||
|
||||
**Goal:** Support importing custom field values from CSV columns. Custom fields should exist in Mila before import for best results.
|
||||
|
||||
**Important Requirements:**
|
||||
- **Custom fields should be created in Mila first** - Unknown custom field columns will be ignored with a warning message
|
||||
- CSV headers for custom fields must match the custom field **name** exactly (same normalization as member fields applies)
|
||||
- Custom field values are validated according to the custom field type (string, integer, boolean, date, email)
|
||||
- Unknown custom field columns (non-existent names) will be ignored with a warning - import continues
|
||||
|
||||
**Tasks:**
|
||||
- [x] Extend `header_mapper.ex` to detect custom field columns by name (using same normalization as member fields)
|
||||
- [x] Query existing custom fields during `prepare/2` to map custom field columns
|
||||
- [x] Collect unknown custom field columns and add warning messages (don't fail import)
|
||||
- [x] Map custom field CSV values to `CustomFieldValue` creation in `process_chunk/4`
|
||||
- [x] Handle custom field type validation (string, integer, boolean, date, email) with proper error messages
|
||||
- [x] Create `CustomFieldValue` records linked to members during import
|
||||
- [x] Validate custom field values and return structured errors with custom field name and reason
|
||||
- [x] UI help text and link to custom field management (implemented in Issue #7)
|
||||
- [x] Update error messages to include custom field validation errors (format: `custom_field: <name> – expected <type>, got: <value>`)
|
||||
- [x] Add UI help text explaining custom field requirements (completed in Issue #7):
|
||||
- "Custom fields must be created in Mila before importing"
|
||||
- "Use the custom field name as the CSV column header (same normalization as member fields)"
|
||||
- Link to custom fields management section
|
||||
- [x] Update CSV templates documentation to explain custom field columns (documented in Issue #1)
|
||||
- [x] Add tests for custom field import (valid, invalid name, type validation, warning for unknown)
|
||||
|
||||
**Definition of Done:**
|
||||
- [x] Custom field columns are recognized by name (with normalization)
|
||||
- [x] Warning messages shown for unknown custom field columns (import continues)
|
||||
- [x] Custom field values are created and linked to members
|
||||
- [x] Type validation works for all custom field types (string, integer, boolean, date, email)
|
||||
- [x] UI clearly explains custom field requirements (completed in Issue #7)
|
||||
- [x] Tests cover custom field import scenarios (including warning for unknown names)
|
||||
- [x] Error messages include custom field validation errors with proper formatting
|
||||
|
||||
**Implementation Notes:**
|
||||
- Custom field lookup is built in `prepare/2` and passed via `custom_field_lookup` in opts
|
||||
- Custom field values are formatted according to type in `format_custom_field_value/2`
|
||||
- Unknown custom field columns generate warnings in `import_state.warnings`
|
||||
|
||||
---
|
||||
|
||||
## Rollout & Risks
|
||||
|
||||
### Rollout Strategy
|
||||
- Dev → Staging → Production (with anonymized real-world CSV tests)
|
||||
|
||||
### Risks & Mitigations
|
||||
|
||||
| Risk | Impact | Likelihood | Mitigation |
|
||||
|---|---:|---:|---|
|
||||
| Large import timeout | High | Medium | 10 MB + 1,000 rows, chunking via `handle_info` |
|
||||
| Encoding issues | Medium | Medium | BOM stripping, templates with BOM |
|
||||
| Invalid CSV format | Medium | High | Clear errors + templates |
|
||||
| Duplicate emails | Low | High | Ash constraint error -> user-friendly message |
|
||||
| Performance (no background jobs) | Medium | Low | Small limits, sequential chunk processing |
|
||||
| Admin access bypass | High | Low | Event-level auth + UI hiding |
|
||||
| Data corruption | High | Low | Per-row validation + best-effort |
|
||||
|
||||
---
|
||||
|
||||
## Appendix
|
||||
|
||||
### Module File Structure
|
||||
|
||||
```
|
||||
lib/
|
||||
├── mv/
|
||||
│ └── membership/
|
||||
│ └── import/
|
||||
│ ├── member_csv.ex # prepare + process_chunk
|
||||
│ ├── import_runner.ex # orchestration: file read, progress merge, chunk process, error format
|
||||
│ ├── csv_parser.ex # delimiter detection + parsing + BOM handling
|
||||
│ └── header_mapper.ex # normalization + header mapping
|
||||
└── mv_web/
|
||||
└── live/
|
||||
├── import_export_live.ex # mount / handle_event / handle_info + glue only
|
||||
└── import_export_live/
|
||||
└── components.ex # UI: custom_fields_notice, template_links, import_form, import_progress, import_results
|
||||
|
||||
priv/
|
||||
└── static/
|
||||
└── templates/
|
||||
├── member_import_en.csv
|
||||
└── member_import_de.csv
|
||||
|
||||
test/
|
||||
├── mv/
|
||||
│ └── membership/
|
||||
│ └── import/
|
||||
│ ├── member_csv_test.exs
|
||||
│ ├── csv_parser_test.exs
|
||||
│ └── header_mapper_test.exs
|
||||
└── fixtures/
|
||||
├── member_import_en.csv
|
||||
├── member_import_de.csv
|
||||
├── member_import_invalid.csv
|
||||
├── member_import_large.csv
|
||||
└── member_import_empty_lines.csv
|
||||
```
|
||||
|
||||
### Example Usage (LiveView)
|
||||
|
||||
```elixir
|
||||
def handle_event("start_import", _params, socket) do
|
||||
assert_admin!(socket.assigns.current_user)
|
||||
|
||||
[{_name, content}] =
|
||||
consume_uploaded_entries(socket, :csv_file, fn %{path: path}, _entry ->
|
||||
{:ok, File.read!(path)}
|
||||
end)
|
||||
|
||||
case Mv.Membership.Import.MemberCSV.prepare(content) do
|
||||
{:ok, import_state} ->
|
||||
socket =
|
||||
socket
|
||||
|> assign(:import_state, import_state)
|
||||
|> assign(:import_progress, %{processed: 0, inserted: 0, failed: 0, errors: []})
|
||||
|> assign(:importing?, true)
|
||||
|
||||
send(self(), {:process_chunk, 0})
|
||||
{:noreply, socket}
|
||||
|
||||
{:error, reason} ->
|
||||
{:noreply, put_flash(socket, :error, reason)}
|
||||
end
|
||||
end
|
||||
|
||||
def handle_info({:process_chunk, idx}, socket) do
|
||||
%{chunks: chunks, column_map: column_map} = socket.assigns.import_state
|
||||
|
||||
case Enum.at(chunks, idx) do
|
||||
nil ->
|
||||
{:noreply, assign(socket, importing?: false)}
|
||||
|
||||
chunk_rows_with_lines ->
|
||||
{:ok, chunk_result} =
|
||||
Mv.Membership.Import.MemberCSV.process_chunk(chunk_rows_with_lines, column_map)
|
||||
|
||||
socket = merge_progress(socket, chunk_result) # caps errors at 50 overall
|
||||
|
||||
send(self(), {:process_chunk, idx + 1})
|
||||
{:noreply, socket}
|
||||
end
|
||||
end
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**End of Implementation Plan**
|
||||
| `email` (required) | email, e-mail, e_mail, mail, e-mail-adresse / E-Mail | |
|
||||
| `first_name` | first name, firstname / Vorname | |
|
||||
| `last_name` | last name, lastname, surname / Nachname, Familienname | |
|
||||
| `join_date` | join date / Beitrittsdatum | ISO-8601 date |
|
||||
| `exit_date` | exit date / Austrittsdatum | ISO-8601 date |
|
||||
| `notes` | notes / Notizen, Bemerkungen | |
|
||||
| `street` | street, address / Straße, Strasse | |
|
||||
| `house_number` | house number, house no / Hausnummer, Nr, Nr., Nummer | |
|
||||
| `postal_code` | postal code, zip, postcode / PLZ, Postleitzahl | |
|
||||
| `city` | city, town / Stadt, Ort | |
|
||||
| `country` | country / Land, Staat | |
|
||||
| `membership_fee_start_date` | membership fee start date, fee start / Beitragsbeginn | ISO-8601 date |
|
||||
|
||||
### Special relationship columns
|
||||
|
||||
- **groups** (headers `Groups` / `Gruppen` / `Gruppe`) — comma-separated group names. Names
|
||||
matched case-insensitively against existing groups; **missing groups are auto-created** during
|
||||
processing. A group-assignment failure fails that row (the member was already created).
|
||||
- **membership_fee_type** (headers `Fee Type`, `fee_type`, `membership_fee_type` / `Beitragsart`)
|
||||
— name matched to an existing `MembershipFeeType`. **Empty cell → default fee type** (no warning).
|
||||
**Matched name → that fee type.** **Unmatched name → default fee type + warning** naming the value.
|
||||
|
||||
These columns are resolved against the DB read-only in `prepare` (`ColumnResolver`) for the
|
||||
preview; the actual writes happen in `process_chunk`.
|
||||
|
||||
### Fields not importable (explicitly ignored)
|
||||
|
||||
- **membership_fee_status** — computed from fee cycles; not stored. Fee-status header variants
|
||||
(`Membership Fee Status`, `Bezahlstatus`, `Mitgliedsbeitragsstatus`) and the DE export label
|
||||
`Startdatum Mitgliedsbeitrag` are placed in the `ignored` list and never mapped. (The UI notice
|
||||
names `Groups`/`Gruppen`, `Fee Type`/`Beitragsart`, and the always-ignored `Bezahlstatus`.)
|
||||
|
||||
## Custom Fields
|
||||
|
||||
- Custom field columns are matched by the custom field **name** (not slug), using the same
|
||||
normalization. Member fields take priority on a name collision.
|
||||
- **Custom fields must exist in Mila before import.** Unknown custom-field columns are ignored
|
||||
with a warning; the import still runs.
|
||||
- Empty custom-field cells create no value. Values are trimmed; type-validated per the custom
|
||||
field's `value_type`:
|
||||
- **string** — any text (trimmed).
|
||||
- **integer** — must parse fully (`Integer.parse` with no remainder); e.g. `42`, `-10`.
|
||||
- **boolean** — case-insensitive `true/false`, `1/0`, `yes/no`, `ja/nein`.
|
||||
- **date** — ISO-8601 `YYYY-MM-DD`.
|
||||
- **email** — validated with `EctoCommons.EmailValidator` (same checks as member email).
|
||||
- A value failing type validation fails the row. Error message format:
|
||||
`custom_field: <name> – expected <type>, got: <value>` (type label is the human-readable
|
||||
`FieldTypes.label/1`, with format hints for boolean/date).
|
||||
|
||||
## Validation & Member Creation (`process_chunk/4` → `process_row`)
|
||||
|
||||
Per row: validate → create member → create custom-field values → assign groups. Sequential.
|
||||
|
||||
- **Email** is required and format-validated (`EctoCommons.EmailValidator`, `Mv.Constants.email_validator_checks()`) on a trimmed value. All string member values are trimmed.
|
||||
- **Date fields** (`join_date`, `exit_date`, `membership_fee_start_date`): empty/blank strings are converted to `nil` so Ash accepts them.
|
||||
- Member created via `Mv.Membership.create_member/2`. Custom field values are passed as
|
||||
`custom_field_values` (Ash union `_union_type`/`_union_value` format), omitted when none.
|
||||
- **Errors** are `%MemberCSV.Error{csv_line_number, field, message}`:
|
||||
- `csv_line_number` is the physical line (1-based); never recomputed in this layer.
|
||||
- Validation errors get `field: :email`; Ash errors prefer the field-level error.
|
||||
- **Duplicate email** (unique constraint) is surfaced as a friendly
|
||||
`"email <addr> has already been taken"` message.
|
||||
- **Error capping** (`max_errors`, default 50, tracked across chunks via `existing_error_count`):
|
||||
once the cap is hit, no further errors are collected but **all rows are still processed** and
|
||||
the `failed` count stays accurate; `errors_truncated?` is set and the UI shows a truncation notice.
|
||||
|
||||
## Templates (`ImportTemplateController`)
|
||||
|
||||
- Generated on the fly (not static files), gated by `can?(:create, Member)`.
|
||||
- One header row: standard member columns (localized EN/DE) + `Groups`/`Gruppen` +
|
||||
`Fee Type`/`Beitragsart` + **every existing custom field name** appended, then one example row.
|
||||
- Semicolon-delimited, RFC-4180 quoting; fields run through `MembersCSV.safe_cell/1` to
|
||||
neutralize spreadsheet formula injection (e.g. a custom-field name like `=HYPERLINK(...)`).
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue