docs: adds higher priority to custom field import
All checks were successful
continuous-integration/drone/push Build is passing

This commit is contained in:
carla 2026-01-07 09:27:03 +01:00
parent 00ff2fa195
commit e9ee4ce21b

View file

@ -29,15 +29,13 @@ A **basic CSV member import feature** that allows administrators to upload a CSV
- Upload CSV file via LiveView file upload - Upload CSV file via LiveView file upload
- Parse CSV with bilingual header support for core member fields (English/German) - Parse CSV with bilingual header support for core member fields (English/German)
- Auto-detect delimiter (`;` or `,`) using header recognition - Auto-detect delimiter (`;` or `,`) using header recognition
- Map CSV columns to core member fields (`first_name`, `last_name`, `email`, `phone_number`, `street`, `postal_code`, `city`) - Map CSV columns to core member fields (`first_name`, `last_name`, `email`, `street`, `postal_code`, `city`)
- Validate each row (required fields: `first_name`, `last_name`, `email`) - **Import custom field values** - Map CSV columns to existing custom fields by name (unknown custom field columns will be ignored with a warning)
- Validate each row (required field: `email`)
- Create members via Ash resource (one-by-one, **no background jobs**, processed in chunks of 200 rows via LiveView messages) - Create members via Ash resource (one-by-one, **no background jobs**, processed in chunks of 200 rows via LiveView messages)
- Display import results: success count, error count, and error details - Display import results: success count, error count, and error details
- Provide static CSV templates (EN/DE) - Provide static CSV templates (EN/DE)
**Optional Enhancement (v1.1 - Last Issue):**
- Custom field import (if time permits, otherwise defer to v2)
**Key Constraints (v1):** **Key Constraints (v1):**
- ✅ **Admin-only feature** - ✅ **Admin-only feature**
- ✅ **No upsert** (create only) - ✅ **No upsert** (create only)
@ -57,9 +55,7 @@ A **basic CSV member import feature** that allows administrators to upload a CSV
- ❌ Background job processing (Oban/GenStage) - ❌ Background job processing (Oban/GenStage)
- ❌ Transactional all-or-nothing import - ❌ Transactional all-or-nothing import
- ❌ Error CSV export/download - ❌ Error CSV export/download
- ⚠️ Custom field import (optional, last issue - defer to v2 if scope is tight)
- ❌ Batch validation preview before import - ❌ Batch validation preview before import
- ❌ Date/boolean field parsing
- ❌ Dynamic template generation - ❌ Dynamic template generation
- ❌ Import history/audit log - ❌ Import history/audit log
- ❌ Import templates for other entities - ❌ Import templates for other entities
@ -79,15 +75,21 @@ A **basic CSV member import feature** that allows administrators to upload a CSV
1. **Navigate to Global Settings** 1. **Navigate to Global Settings**
2. **Access Import Section** 2. **Access Import Section**
- **Important notice:** Custom fields should be created in Mila before importing CSV files with custom field columns (unknown columns will be ignored with a warning)
- Upload area (drag & drop or file picker) - Upload area (drag & drop or file picker)
- Template download links (English / German) - Template download links (English / German)
- Help text explaining CSV format - Help text explaining CSV format and custom field requirements
3. **Download Template (Optional)** 3. **Ensure Custom Fields Exist (if importing custom fields)**
4. **Prepare CSV File** - Navigate to Custom Fields section and create required custom fields
5. **Upload CSV** - Note the name/identifier for each custom field (used as CSV header)
6. **Start Import** 4. **Download Template (Optional)**
5. **Prepare CSV File**
- Include custom field columns using the custom field name as header (e.g., `membership_number`, `birth_date`)
6. **Upload CSV**
7. **Start Import**
- Runs server-side via LiveView messages (may take up to ~30 seconds for large files) - Runs server-side via LiveView messages (may take up to ~30 seconds for large files)
7. **View Results** - Warning messages if custom field columns reference non-existent custom fields (columns will be ignored)
8. **View Results**
- Success count - Success count
- Error count - Error count
- First 50 errors, each with: - First 50 errors, each with:
@ -126,15 +128,21 @@ A **basic CSV member import feature** that allows administrators to upload a CSV
### Column Headers ### Column Headers
**v1 Supported Fields (Core Member Fields Only):** **v1 Supported Fields:**
- `first_name` / `Vorname` (required)
- `last_name` / `Nachname` (required) **Core Member Fields:**
- `first_name` / `Vorname` (optional)
- `last_name` / `Nachname` (optional)
- `email` / `E-Mail` (required) - `email` / `E-Mail` (required)
- `phone_number` / `Telefon` (optional)
- `street` / `Straße` (optional) - `street` / `Straße` (optional)
- `postal_code` / `PLZ` / `Postleitzahl` (optional) - `postal_code` / `PLZ` / `Postleitzahl` (optional)
- `city` / `Stadt` (optional) - `city` / `Stadt` (optional)
**Custom Fields:**
- Any custom field column using the custom field's **name** as the header (e.g., `membership_number`, `birth_date`)
- **Important:** Custom fields must be created in Mila before importing. The CSV header must match the custom field name exactly (same normalization as member fields).
- **Behavior:** If the CSV contains custom field columns that don't exist in Mila, a warning message will be shown and those columns will be ignored during import.
**Member Field Header Mapping:** **Member Field Header Mapping:**
| Canonical Field | English Variants | German Variants | | Canonical Field | English Variants | German Variants |
@ -142,7 +150,6 @@ A **basic CSV member import feature** that allows administrators to upload a CSV
| `first_name` | `first_name`, `firstname` | `Vorname`, `vorname` | | `first_name` | `first_name`, `firstname` | `Vorname`, `vorname` |
| `last_name` | `last_name`, `lastname`, `surname` | `Nachname`, `nachname`, `Familienname` | | `last_name` | `last_name`, `lastname`, `surname` | `Nachname`, `nachname`, `Familienname` |
| `email` | `email`, `e-mail`, `e_mail` | `E-Mail`, `e-mail`, `e_mail` | | `email` | `email`, `e-mail`, `e_mail` | `E-Mail`, `e-mail`, `e_mail` |
| `phone_number` | `phone_number`, `phone`, `telephone` | `Telefon`, `telefon` |
| `street` | `street`, `address` | `Straße`, `strasse`, `Strasse` | | `street` | `street`, `address` | `Straße`, `strasse`, `Strasse` |
| `postal_code` | `postal_code`, `zip`, `postcode` | `PLZ`, `plz`, `Postleitzahl`, `postleitzahl` | | `postal_code` | `postal_code`, `zip`, `postcode` | `PLZ`, `plz`, `Postleitzahl`, `postleitzahl` |
| `city` | `city`, `town` | `Stadt`, `stadt`, `Ort` | | `city` | `city`, `town` | `Stadt`, `stadt`, `Ort` |
@ -157,7 +164,12 @@ A **basic CSV member import feature** that allows administrators to upload a CSV
**Unknown columns:** ignored (no error) **Unknown columns:** ignored (no error)
**Required fields:** `first_name`, `last_name`, `email` **Required fields:** `email`
**Custom Field Columns:**
- Custom field columns are identified by matching the normalized CSV header to the custom field `name` (not slug)
- Same normalization rules apply as for member fields (trim, lowercase, Unicode normalization, underscore replacement)
- Unknown custom field columns (non-existent names) will be ignored with a warning message
### CSV Template Files ### CSV Template Files
@ -167,6 +179,7 @@ A **basic CSV member import feature** that allows administrators to upload a CSV
**Content:** **Content:**
- Header row with required + common optional fields - Header row with required + common optional fields
- **Note:** Custom field columns are not included in templates by default (users add them based on their custom field configuration)
- One example row - One example row
- Uses semicolon delimiter (`;`) - Uses semicolon delimiter (`;`)
- UTF-8 encoding **with BOM** (Excel compatibility) - UTF-8 encoding **with BOM** (Excel compatibility)
@ -223,9 +236,9 @@ A **basic CSV member import feature** that allows administrators to upload a CSV
### Module Structure ### Module Structure
**New Modules:** **New Modules:**
- `lib/mv/membership/import/member_csv.ex` - import orchestration + chunk processing - `lib/mv/membership/import/member_csv.ex` - import orchestration + chunk processing + custom field handling
- `lib/mv/membership/import/csv_parser.ex` - delimiter detection + parsing + BOM handling - `lib/mv/membership/import/csv_parser.ex` - delimiter detection + parsing + BOM handling
- `lib/mv/membership/import/header_mapper.ex` - normalization + header mapping - `lib/mv/membership/import/header_mapper.ex` - normalization + header mapping (core fields + custom fields)
**Modified Modules:** **Modified Modules:**
- `lib/mv_web/live/global_settings_live.ex` - render import section, handle upload/events/messages - `lib/mv_web/live/global_settings_live.ex` - render import section, handle upload/events/messages
@ -238,12 +251,14 @@ A **basic CSV member import feature** that allows administrators to upload a CSV
- Strip BOM - Strip BOM
- Detect delimiter (header recognition) - Detect delimiter (header recognition)
- Parse header + rows - Parse header + rows
- Map headers to canonical fields - Map headers to canonical fields (core member fields)
- **Query existing custom fields and map custom field columns by name** (using same normalization as member fields)
- **Warn about unknown custom field columns** (non-existent names will be ignored with warning)
- Early abort if required headers missing - Early abort if required headers missing
- Row count check - Row count check
- Return `import_state` containing chunks and metadata - Return `import_state` containing chunks, column_map, and custom_field_map
4. **Process:** LiveView drives chunk processing via `handle_info` 4. **Process:** LiveView drives chunk processing via `handle_info`
- For each chunk: validate + create + collect errors - For each chunk: validate + create member + create custom field values + collect errors
5. **Results:** LiveView shows progress + final summary 5. **Results:** LiveView shows progress + final summary
### Types & Key Consistency ### Types & Key Consistency
@ -368,8 +383,9 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c
- [ ] Implement `normalize_header/1` - [ ] Implement `normalize_header/1`
- [ ] Normalize mapping variants once and compare normalized strings - [ ] Normalize mapping variants once and compare normalized strings
- [ ] Build `column_map` (canonical field -> column index) - [ ] Build `column_map` (canonical field -> column index)
- [ ] **Early abort if required headers missing** (`first_name`, `last_name`, `email`) - [ ] **Early abort if required headers missing** (`email`)
- [ ] Ignore unknown columns - [ ] Ignore unknown columns (member fields only)
- [ ] **Separate custom field column detection** (by name, with normalization)
**Definition of Done:** **Definition of Done:**
- [ ] English/German headers map correctly - [ ] English/German headers map correctly
@ -385,7 +401,7 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c
**Tasks:** **Tasks:**
- [ ] Implement `validate_row/3 (row_map, csv_line_number, opts)` - [ ] Implement `validate_row/3 (row_map, csv_line_number, opts)`
- [ ] Required field presence (`first_name`, `last_name`, `email`) - [ ] Required field presence (`email`)
- [ ] Email format validation (EctoCommons.EmailValidator) - [ ] Email format validation (EctoCommons.EmailValidator)
- [ ] Trim values before validation - [ ] Trim values before validation
- [ ] Gettext-backed error messages - [ ] Gettext-backed error messages
@ -421,6 +437,10 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c
**Tasks:** **Tasks:**
- [ ] Render import section only for admins - [ ] Render import section only for admins
- [ ] **Add prominent UI notice about custom fields:**
- Display alert/info box: "Custom fields must be created in Mila before importing CSV files with custom field columns"
- Explain: "Use the custom field name as the CSV column header (same normalization as member fields applies)"
- Add link to custom fields management section
- [ ] Configure `allow_upload/3`: - [ ] Configure `allow_upload/3`:
- `.csv` only, `max_entries: 1`, `max_file_size: 10MB`, `auto_upload: false` - `.csv` only, `max_entries: 1`, `max_file_size: 10MB`, `auto_upload: false`
- [ ] `handle_event("start_import", ...)`: - [ ] `handle_event("start_import", ...)`:
@ -439,6 +459,7 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c
- Success count - Success count
- Failure count - Failure count
- Error list (line number + message + field) - Error list (line number + message + field)
- **Warning messages for unknown custom field columns** (non-existent names) shown in results
**Template links:** **Template links:**
- Link `/templates/member_import_en.csv` and `/templates/member_import_de.csv` via Phoenix static path helpers. - Link `/templates/member_import_en.csv` and `/templates/member_import_de.csv` via Phoenix static path helpers.
@ -471,8 +492,10 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c
**Tasks:** **Tasks:**
- [ ] Fixtures: - [ ] Fixtures:
- valid EN/DE - valid EN/DE (core fields only)
- valid with custom fields
- invalid - invalid
- unknown custom field name (non-existent, should show warning)
- too many rows (1,001) - too many rows (1,001)
- BOM + `;` delimiter fixture - BOM + `;` delimiter fixture
- fixture with empty line(s) to validate correct line numbers - fixture with empty line(s) to validate correct line numbers
@ -481,6 +504,8 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c
- upload + start import - upload + start import
- success + error rendering - success + error rendering
- row limit + file size errors - row limit + file size errors
- custom field import success
- custom field import warning (non-existent name, column ignored)
--- ---
@ -495,12 +520,42 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c
--- ---
### Issue #11: Custom Field Import (Optional - v1.1) ### Issue #11: Custom Field Import
**Dependencies:** Issue #10 **Dependencies:** Issue #6 (Persistence)
**Status:** Optional
*(unchanged — intentionally deferred)* **Priority:** High (Core v1 Feature)
**Goal:** Support importing custom field values from CSV columns. Custom fields should exist in Mila before import for best results.
**Important Requirements:**
- **Custom fields should be created in Mila first** - Unknown custom field columns will be ignored with a warning message
- CSV headers for custom fields must match the custom field **name** exactly (same normalization as member fields applies)
- Custom field values are validated according to the custom field type (string, integer, boolean, date, email)
- Unknown custom field columns (non-existent names) will be ignored with a warning - import continues
**Tasks:**
- [ ] Extend `header_mapper.ex` to detect custom field columns by name (using same normalization as member fields)
- [ ] Query existing custom fields during `prepare/2` to map custom field columns
- [ ] Collect unknown custom field columns and add warning messages (don't fail import)
- [ ] Map custom field CSV values to `CustomFieldValue` creation in `process_chunk/3`
- [ ] Handle custom field type validation (string, integer, boolean, date, email)
- [ ] Create `CustomFieldValue` records linked to members during import
- [ ] Update error messages to include custom field validation errors
- [ ] Add UI help text explaining custom field requirements:
- "Custom fields must be created in Mila before importing"
- "Use the custom field name as the CSV column header (same normalization as member fields)"
- Link to custom fields management section
- [ ] Update CSV templates documentation to explain custom field columns
- [ ] Add tests for custom field import (valid, invalid name, type validation, warning for unknown)
**Definition of Done:**
- [ ] Custom field columns are recognized by name (with normalization)
- [ ] Warning messages shown for unknown custom field columns (import continues)
- [ ] Custom field values are created and linked to members
- [ ] Type validation works for all custom field types
- [ ] UI clearly explains custom field requirements
- [ ] Tests cover custom field import scenarios (including warning for unknown names)
--- ---