From e9ee4ce21b8c986e2caf1e6962d4d3caa0338a0e Mon Sep 17 00:00:00 2001 From: carla Date: Wed, 7 Jan 2026 09:27:03 +0100 Subject: [PATCH] docs: adds higher priority to custom field import --- docs/csv-member-import-v1.md | 119 +++++++++++++++++++++++++---------- 1 file changed, 87 insertions(+), 32 deletions(-) diff --git a/docs/csv-member-import-v1.md b/docs/csv-member-import-v1.md index 30409b8..2bdbe69 100644 --- a/docs/csv-member-import-v1.md +++ b/docs/csv-member-import-v1.md @@ -29,15 +29,13 @@ A **basic CSV member import feature** that allows administrators to upload a CSV - Upload CSV file via LiveView file upload - Parse CSV with bilingual header support for core member fields (English/German) - Auto-detect delimiter (`;` or `,`) using header recognition -- Map CSV columns to core member fields (`first_name`, `last_name`, `email`, `phone_number`, `street`, `postal_code`, `city`) -- Validate each row (required fields: `first_name`, `last_name`, `email`) +- Map CSV columns to core member fields (`first_name`, `last_name`, `email`, `street`, `postal_code`, `city`) +- **Import custom field values** - Map CSV columns to existing custom fields by name (unknown custom field columns will be ignored with a warning) +- Validate each row (required field: `email`) - Create members via Ash resource (one-by-one, **no background jobs**, processed in chunks of 200 rows via LiveView messages) - Display import results: success count, error count, and error details - Provide static CSV templates (EN/DE) -**Optional Enhancement (v1.1 - Last Issue):** -- Custom field import (if time permits, otherwise defer to v2) - **Key Constraints (v1):** - ✅ **Admin-only feature** - ✅ **No upsert** (create only) @@ -57,9 +55,7 @@ A **basic CSV member import feature** that allows administrators to upload a CSV - ❌ Background job processing (Oban/GenStage) - ❌ Transactional all-or-nothing import - ❌ Error CSV export/download -- ⚠️ Custom field import (optional, last issue - defer to v2 if scope is tight) - ❌ Batch validation preview before import -- ❌ Date/boolean field parsing - ❌ Dynamic template generation - ❌ Import history/audit log - ❌ Import templates for other entities @@ -79,15 +75,21 @@ A **basic CSV member import feature** that allows administrators to upload a CSV 1. **Navigate to Global Settings** 2. **Access Import Section** + - **Important notice:** Custom fields should be created in Mila before importing CSV files with custom field columns (unknown columns will be ignored with a warning) - Upload area (drag & drop or file picker) - Template download links (English / German) - - Help text explaining CSV format -3. **Download Template (Optional)** -4. **Prepare CSV File** -5. **Upload CSV** -6. **Start Import** + - Help text explaining CSV format and custom field requirements +3. **Ensure Custom Fields Exist (if importing custom fields)** + - Navigate to Custom Fields section and create required custom fields + - Note the name/identifier for each custom field (used as CSV header) +4. **Download Template (Optional)** +5. **Prepare CSV File** + - Include custom field columns using the custom field name as header (e.g., `membership_number`, `birth_date`) +6. **Upload CSV** +7. **Start Import** - Runs server-side via LiveView messages (may take up to ~30 seconds for large files) -7. **View Results** + - Warning messages if custom field columns reference non-existent custom fields (columns will be ignored) +8. **View Results** - Success count - Error count - First 50 errors, each with: @@ -126,15 +128,21 @@ A **basic CSV member import feature** that allows administrators to upload a CSV ### Column Headers -**v1 Supported Fields (Core Member Fields Only):** -- `first_name` / `Vorname` (required) -- `last_name` / `Nachname` (required) +**v1 Supported Fields:** + +**Core Member Fields:** +- `first_name` / `Vorname` (optional) +- `last_name` / `Nachname` (optional) - `email` / `E-Mail` (required) -- `phone_number` / `Telefon` (optional) - `street` / `Straße` (optional) - `postal_code` / `PLZ` / `Postleitzahl` (optional) - `city` / `Stadt` (optional) +**Custom Fields:** +- Any custom field column using the custom field's **name** as the header (e.g., `membership_number`, `birth_date`) +- **Important:** Custom fields must be created in Mila before importing. The CSV header must match the custom field name exactly (same normalization as member fields). +- **Behavior:** If the CSV contains custom field columns that don't exist in Mila, a warning message will be shown and those columns will be ignored during import. + **Member Field Header Mapping:** | Canonical Field | English Variants | German Variants | @@ -142,7 +150,6 @@ A **basic CSV member import feature** that allows administrators to upload a CSV | `first_name` | `first_name`, `firstname` | `Vorname`, `vorname` | | `last_name` | `last_name`, `lastname`, `surname` | `Nachname`, `nachname`, `Familienname` | | `email` | `email`, `e-mail`, `e_mail` | `E-Mail`, `e-mail`, `e_mail` | -| `phone_number` | `phone_number`, `phone`, `telephone` | `Telefon`, `telefon` | | `street` | `street`, `address` | `Straße`, `strasse`, `Strasse` | | `postal_code` | `postal_code`, `zip`, `postcode` | `PLZ`, `plz`, `Postleitzahl`, `postleitzahl` | | `city` | `city`, `town` | `Stadt`, `stadt`, `Ort` | @@ -157,7 +164,12 @@ A **basic CSV member import feature** that allows administrators to upload a CSV **Unknown columns:** ignored (no error) -**Required fields:** `first_name`, `last_name`, `email` +**Required fields:** `email` + +**Custom Field Columns:** +- Custom field columns are identified by matching the normalized CSV header to the custom field `name` (not slug) +- Same normalization rules apply as for member fields (trim, lowercase, Unicode normalization, underscore replacement) +- Unknown custom field columns (non-existent names) will be ignored with a warning message ### CSV Template Files @@ -167,6 +179,7 @@ A **basic CSV member import feature** that allows administrators to upload a CSV **Content:** - Header row with required + common optional fields +- **Note:** Custom field columns are not included in templates by default (users add them based on their custom field configuration) - One example row - Uses semicolon delimiter (`;`) - UTF-8 encoding **with BOM** (Excel compatibility) @@ -223,9 +236,9 @@ A **basic CSV member import feature** that allows administrators to upload a CSV ### Module Structure **New Modules:** -- `lib/mv/membership/import/member_csv.ex` - import orchestration + chunk processing +- `lib/mv/membership/import/member_csv.ex` - import orchestration + chunk processing + custom field handling - `lib/mv/membership/import/csv_parser.ex` - delimiter detection + parsing + BOM handling -- `lib/mv/membership/import/header_mapper.ex` - normalization + header mapping +- `lib/mv/membership/import/header_mapper.ex` - normalization + header mapping (core fields + custom fields) **Modified Modules:** - `lib/mv_web/live/global_settings_live.ex` - render import section, handle upload/events/messages @@ -238,12 +251,14 @@ A **basic CSV member import feature** that allows administrators to upload a CSV - Strip BOM - Detect delimiter (header recognition) - Parse header + rows - - Map headers to canonical fields + - Map headers to canonical fields (core member fields) + - **Query existing custom fields and map custom field columns by name** (using same normalization as member fields) + - **Warn about unknown custom field columns** (non-existent names will be ignored with warning) - Early abort if required headers missing - Row count check - - Return `import_state` containing chunks and metadata + - Return `import_state` containing chunks, column_map, and custom_field_map 4. **Process:** LiveView drives chunk processing via `handle_info` - - For each chunk: validate + create + collect errors + - For each chunk: validate + create member + create custom field values + collect errors 5. **Results:** LiveView shows progress + final summary ### Types & Key Consistency @@ -368,8 +383,9 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c - [ ] Implement `normalize_header/1` - [ ] Normalize mapping variants once and compare normalized strings - [ ] Build `column_map` (canonical field -> column index) -- [ ] **Early abort if required headers missing** (`first_name`, `last_name`, `email`) -- [ ] Ignore unknown columns +- [ ] **Early abort if required headers missing** (`email`) +- [ ] Ignore unknown columns (member fields only) +- [ ] **Separate custom field column detection** (by name, with normalization) **Definition of Done:** - [ ] English/German headers map correctly @@ -385,7 +401,7 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c **Tasks:** - [ ] Implement `validate_row/3 (row_map, csv_line_number, opts)` -- [ ] Required field presence (`first_name`, `last_name`, `email`) +- [ ] Required field presence (`email`) - [ ] Email format validation (EctoCommons.EmailValidator) - [ ] Trim values before validation - [ ] Gettext-backed error messages @@ -421,6 +437,10 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c **Tasks:** - [ ] Render import section only for admins +- [ ] **Add prominent UI notice about custom fields:** + - Display alert/info box: "Custom fields must be created in Mila before importing CSV files with custom field columns" + - Explain: "Use the custom field name as the CSV column header (same normalization as member fields applies)" + - Add link to custom fields management section - [ ] Configure `allow_upload/3`: - `.csv` only, `max_entries: 1`, `max_file_size: 10MB`, `auto_upload: false` - [ ] `handle_event("start_import", ...)`: @@ -439,6 +459,7 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c - Success count - Failure count - Error list (line number + message + field) + - **Warning messages for unknown custom field columns** (non-existent names) shown in results **Template links:** - Link `/templates/member_import_en.csv` and `/templates/member_import_de.csv` via Phoenix static path helpers. @@ -471,8 +492,10 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c **Tasks:** - [ ] Fixtures: - - valid EN/DE + - valid EN/DE (core fields only) + - valid with custom fields - invalid + - unknown custom field name (non-existent, should show warning) - too many rows (1,001) - BOM + `;` delimiter fixture - fixture with empty line(s) to validate correct line numbers @@ -481,6 +504,8 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c - upload + start import - success + error rendering - row limit + file size errors + - custom field import success + - custom field import warning (non-existent name, column ignored) --- @@ -495,12 +520,42 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c --- -### Issue #11: Custom Field Import (Optional - v1.1) +### Issue #11: Custom Field Import -**Dependencies:** Issue #10 -**Status:** Optional +**Dependencies:** Issue #6 (Persistence) -*(unchanged — intentionally deferred)* +**Priority:** High (Core v1 Feature) + +**Goal:** Support importing custom field values from CSV columns. Custom fields should exist in Mila before import for best results. + +**Important Requirements:** +- **Custom fields should be created in Mila first** - Unknown custom field columns will be ignored with a warning message +- CSV headers for custom fields must match the custom field **name** exactly (same normalization as member fields applies) +- Custom field values are validated according to the custom field type (string, integer, boolean, date, email) +- Unknown custom field columns (non-existent names) will be ignored with a warning - import continues + +**Tasks:** +- [ ] Extend `header_mapper.ex` to detect custom field columns by name (using same normalization as member fields) +- [ ] Query existing custom fields during `prepare/2` to map custom field columns +- [ ] Collect unknown custom field columns and add warning messages (don't fail import) +- [ ] Map custom field CSV values to `CustomFieldValue` creation in `process_chunk/3` +- [ ] Handle custom field type validation (string, integer, boolean, date, email) +- [ ] Create `CustomFieldValue` records linked to members during import +- [ ] Update error messages to include custom field validation errors +- [ ] Add UI help text explaining custom field requirements: + - "Custom fields must be created in Mila before importing" + - "Use the custom field name as the CSV column header (same normalization as member fields)" + - Link to custom fields management section +- [ ] Update CSV templates documentation to explain custom field columns +- [ ] Add tests for custom field import (valid, invalid name, type validation, warning for unknown) + +**Definition of Done:** +- [ ] Custom field columns are recognized by name (with normalization) +- [ ] Warning messages shown for unknown custom field columns (import continues) +- [ ] Custom field values are created and linked to members +- [ ] Type validation works for all custom field types +- [ ] UI clearly explains custom field requirements +- [ ] Tests cover custom field import scenarios (including warning for unknown names) ---