docs: adds higher priority to custom field import
All checks were successful
continuous-integration/drone/push Build is passing
All checks were successful
continuous-integration/drone/push Build is passing
This commit is contained in:
parent
00ff2fa195
commit
e9ee4ce21b
1 changed files with 87 additions and 32 deletions
|
|
@ -29,15 +29,13 @@ A **basic CSV member import feature** that allows administrators to upload a CSV
|
||||||
- Upload CSV file via LiveView file upload
|
- Upload CSV file via LiveView file upload
|
||||||
- Parse CSV with bilingual header support for core member fields (English/German)
|
- Parse CSV with bilingual header support for core member fields (English/German)
|
||||||
- Auto-detect delimiter (`;` or `,`) using header recognition
|
- Auto-detect delimiter (`;` or `,`) using header recognition
|
||||||
- Map CSV columns to core member fields (`first_name`, `last_name`, `email`, `phone_number`, `street`, `postal_code`, `city`)
|
- Map CSV columns to core member fields (`first_name`, `last_name`, `email`, `street`, `postal_code`, `city`)
|
||||||
- Validate each row (required fields: `first_name`, `last_name`, `email`)
|
- **Import custom field values** - Map CSV columns to existing custom fields by name (unknown custom field columns will be ignored with a warning)
|
||||||
|
- Validate each row (required field: `email`)
|
||||||
- Create members via Ash resource (one-by-one, **no background jobs**, processed in chunks of 200 rows via LiveView messages)
|
- Create members via Ash resource (one-by-one, **no background jobs**, processed in chunks of 200 rows via LiveView messages)
|
||||||
- Display import results: success count, error count, and error details
|
- Display import results: success count, error count, and error details
|
||||||
- Provide static CSV templates (EN/DE)
|
- Provide static CSV templates (EN/DE)
|
||||||
|
|
||||||
**Optional Enhancement (v1.1 - Last Issue):**
|
|
||||||
- Custom field import (if time permits, otherwise defer to v2)
|
|
||||||
|
|
||||||
**Key Constraints (v1):**
|
**Key Constraints (v1):**
|
||||||
- ✅ **Admin-only feature**
|
- ✅ **Admin-only feature**
|
||||||
- ✅ **No upsert** (create only)
|
- ✅ **No upsert** (create only)
|
||||||
|
|
@ -57,9 +55,7 @@ A **basic CSV member import feature** that allows administrators to upload a CSV
|
||||||
- ❌ Background job processing (Oban/GenStage)
|
- ❌ Background job processing (Oban/GenStage)
|
||||||
- ❌ Transactional all-or-nothing import
|
- ❌ Transactional all-or-nothing import
|
||||||
- ❌ Error CSV export/download
|
- ❌ Error CSV export/download
|
||||||
- ⚠️ Custom field import (optional, last issue - defer to v2 if scope is tight)
|
|
||||||
- ❌ Batch validation preview before import
|
- ❌ Batch validation preview before import
|
||||||
- ❌ Date/boolean field parsing
|
|
||||||
- ❌ Dynamic template generation
|
- ❌ Dynamic template generation
|
||||||
- ❌ Import history/audit log
|
- ❌ Import history/audit log
|
||||||
- ❌ Import templates for other entities
|
- ❌ Import templates for other entities
|
||||||
|
|
@ -79,15 +75,21 @@ A **basic CSV member import feature** that allows administrators to upload a CSV
|
||||||
|
|
||||||
1. **Navigate to Global Settings**
|
1. **Navigate to Global Settings**
|
||||||
2. **Access Import Section**
|
2. **Access Import Section**
|
||||||
|
- **Important notice:** Custom fields should be created in Mila before importing CSV files with custom field columns (unknown columns will be ignored with a warning)
|
||||||
- Upload area (drag & drop or file picker)
|
- Upload area (drag & drop or file picker)
|
||||||
- Template download links (English / German)
|
- Template download links (English / German)
|
||||||
- Help text explaining CSV format
|
- Help text explaining CSV format and custom field requirements
|
||||||
3. **Download Template (Optional)**
|
3. **Ensure Custom Fields Exist (if importing custom fields)**
|
||||||
4. **Prepare CSV File**
|
- Navigate to Custom Fields section and create required custom fields
|
||||||
5. **Upload CSV**
|
- Note the name/identifier for each custom field (used as CSV header)
|
||||||
6. **Start Import**
|
4. **Download Template (Optional)**
|
||||||
|
5. **Prepare CSV File**
|
||||||
|
- Include custom field columns using the custom field name as header (e.g., `membership_number`, `birth_date`)
|
||||||
|
6. **Upload CSV**
|
||||||
|
7. **Start Import**
|
||||||
- Runs server-side via LiveView messages (may take up to ~30 seconds for large files)
|
- Runs server-side via LiveView messages (may take up to ~30 seconds for large files)
|
||||||
7. **View Results**
|
- Warning messages if custom field columns reference non-existent custom fields (columns will be ignored)
|
||||||
|
8. **View Results**
|
||||||
- Success count
|
- Success count
|
||||||
- Error count
|
- Error count
|
||||||
- First 50 errors, each with:
|
- First 50 errors, each with:
|
||||||
|
|
@ -126,15 +128,21 @@ A **basic CSV member import feature** that allows administrators to upload a CSV
|
||||||
|
|
||||||
### Column Headers
|
### Column Headers
|
||||||
|
|
||||||
**v1 Supported Fields (Core Member Fields Only):**
|
**v1 Supported Fields:**
|
||||||
- `first_name` / `Vorname` (required)
|
|
||||||
- `last_name` / `Nachname` (required)
|
**Core Member Fields:**
|
||||||
|
- `first_name` / `Vorname` (optional)
|
||||||
|
- `last_name` / `Nachname` (optional)
|
||||||
- `email` / `E-Mail` (required)
|
- `email` / `E-Mail` (required)
|
||||||
- `phone_number` / `Telefon` (optional)
|
|
||||||
- `street` / `Straße` (optional)
|
- `street` / `Straße` (optional)
|
||||||
- `postal_code` / `PLZ` / `Postleitzahl` (optional)
|
- `postal_code` / `PLZ` / `Postleitzahl` (optional)
|
||||||
- `city` / `Stadt` (optional)
|
- `city` / `Stadt` (optional)
|
||||||
|
|
||||||
|
**Custom Fields:**
|
||||||
|
- Any custom field column using the custom field's **name** as the header (e.g., `membership_number`, `birth_date`)
|
||||||
|
- **Important:** Custom fields must be created in Mila before importing. The CSV header must match the custom field name exactly (same normalization as member fields).
|
||||||
|
- **Behavior:** If the CSV contains custom field columns that don't exist in Mila, a warning message will be shown and those columns will be ignored during import.
|
||||||
|
|
||||||
**Member Field Header Mapping:**
|
**Member Field Header Mapping:**
|
||||||
|
|
||||||
| Canonical Field | English Variants | German Variants |
|
| Canonical Field | English Variants | German Variants |
|
||||||
|
|
@ -142,7 +150,6 @@ A **basic CSV member import feature** that allows administrators to upload a CSV
|
||||||
| `first_name` | `first_name`, `firstname` | `Vorname`, `vorname` |
|
| `first_name` | `first_name`, `firstname` | `Vorname`, `vorname` |
|
||||||
| `last_name` | `last_name`, `lastname`, `surname` | `Nachname`, `nachname`, `Familienname` |
|
| `last_name` | `last_name`, `lastname`, `surname` | `Nachname`, `nachname`, `Familienname` |
|
||||||
| `email` | `email`, `e-mail`, `e_mail` | `E-Mail`, `e-mail`, `e_mail` |
|
| `email` | `email`, `e-mail`, `e_mail` | `E-Mail`, `e-mail`, `e_mail` |
|
||||||
| `phone_number` | `phone_number`, `phone`, `telephone` | `Telefon`, `telefon` |
|
|
||||||
| `street` | `street`, `address` | `Straße`, `strasse`, `Strasse` |
|
| `street` | `street`, `address` | `Straße`, `strasse`, `Strasse` |
|
||||||
| `postal_code` | `postal_code`, `zip`, `postcode` | `PLZ`, `plz`, `Postleitzahl`, `postleitzahl` |
|
| `postal_code` | `postal_code`, `zip`, `postcode` | `PLZ`, `plz`, `Postleitzahl`, `postleitzahl` |
|
||||||
| `city` | `city`, `town` | `Stadt`, `stadt`, `Ort` |
|
| `city` | `city`, `town` | `Stadt`, `stadt`, `Ort` |
|
||||||
|
|
@ -157,7 +164,12 @@ A **basic CSV member import feature** that allows administrators to upload a CSV
|
||||||
|
|
||||||
**Unknown columns:** ignored (no error)
|
**Unknown columns:** ignored (no error)
|
||||||
|
|
||||||
**Required fields:** `first_name`, `last_name`, `email`
|
**Required fields:** `email`
|
||||||
|
|
||||||
|
**Custom Field Columns:**
|
||||||
|
- Custom field columns are identified by matching the normalized CSV header to the custom field `name` (not slug)
|
||||||
|
- Same normalization rules apply as for member fields (trim, lowercase, Unicode normalization, underscore replacement)
|
||||||
|
- Unknown custom field columns (non-existent names) will be ignored with a warning message
|
||||||
|
|
||||||
### CSV Template Files
|
### CSV Template Files
|
||||||
|
|
||||||
|
|
@ -167,6 +179,7 @@ A **basic CSV member import feature** that allows administrators to upload a CSV
|
||||||
|
|
||||||
**Content:**
|
**Content:**
|
||||||
- Header row with required + common optional fields
|
- Header row with required + common optional fields
|
||||||
|
- **Note:** Custom field columns are not included in templates by default (users add them based on their custom field configuration)
|
||||||
- One example row
|
- One example row
|
||||||
- Uses semicolon delimiter (`;`)
|
- Uses semicolon delimiter (`;`)
|
||||||
- UTF-8 encoding **with BOM** (Excel compatibility)
|
- UTF-8 encoding **with BOM** (Excel compatibility)
|
||||||
|
|
@ -223,9 +236,9 @@ A **basic CSV member import feature** that allows administrators to upload a CSV
|
||||||
### Module Structure
|
### Module Structure
|
||||||
|
|
||||||
**New Modules:**
|
**New Modules:**
|
||||||
- `lib/mv/membership/import/member_csv.ex` - import orchestration + chunk processing
|
- `lib/mv/membership/import/member_csv.ex` - import orchestration + chunk processing + custom field handling
|
||||||
- `lib/mv/membership/import/csv_parser.ex` - delimiter detection + parsing + BOM handling
|
- `lib/mv/membership/import/csv_parser.ex` - delimiter detection + parsing + BOM handling
|
||||||
- `lib/mv/membership/import/header_mapper.ex` - normalization + header mapping
|
- `lib/mv/membership/import/header_mapper.ex` - normalization + header mapping (core fields + custom fields)
|
||||||
|
|
||||||
**Modified Modules:**
|
**Modified Modules:**
|
||||||
- `lib/mv_web/live/global_settings_live.ex` - render import section, handle upload/events/messages
|
- `lib/mv_web/live/global_settings_live.ex` - render import section, handle upload/events/messages
|
||||||
|
|
@ -238,12 +251,14 @@ A **basic CSV member import feature** that allows administrators to upload a CSV
|
||||||
- Strip BOM
|
- Strip BOM
|
||||||
- Detect delimiter (header recognition)
|
- Detect delimiter (header recognition)
|
||||||
- Parse header + rows
|
- Parse header + rows
|
||||||
- Map headers to canonical fields
|
- Map headers to canonical fields (core member fields)
|
||||||
|
- **Query existing custom fields and map custom field columns by name** (using same normalization as member fields)
|
||||||
|
- **Warn about unknown custom field columns** (non-existent names will be ignored with warning)
|
||||||
- Early abort if required headers missing
|
- Early abort if required headers missing
|
||||||
- Row count check
|
- Row count check
|
||||||
- Return `import_state` containing chunks and metadata
|
- Return `import_state` containing chunks, column_map, and custom_field_map
|
||||||
4. **Process:** LiveView drives chunk processing via `handle_info`
|
4. **Process:** LiveView drives chunk processing via `handle_info`
|
||||||
- For each chunk: validate + create + collect errors
|
- For each chunk: validate + create member + create custom field values + collect errors
|
||||||
5. **Results:** LiveView shows progress + final summary
|
5. **Results:** LiveView shows progress + final summary
|
||||||
|
|
||||||
### Types & Key Consistency
|
### Types & Key Consistency
|
||||||
|
|
@ -368,8 +383,9 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c
|
||||||
- [ ] Implement `normalize_header/1`
|
- [ ] Implement `normalize_header/1`
|
||||||
- [ ] Normalize mapping variants once and compare normalized strings
|
- [ ] Normalize mapping variants once and compare normalized strings
|
||||||
- [ ] Build `column_map` (canonical field -> column index)
|
- [ ] Build `column_map` (canonical field -> column index)
|
||||||
- [ ] **Early abort if required headers missing** (`first_name`, `last_name`, `email`)
|
- [ ] **Early abort if required headers missing** (`email`)
|
||||||
- [ ] Ignore unknown columns
|
- [ ] Ignore unknown columns (member fields only)
|
||||||
|
- [ ] **Separate custom field column detection** (by name, with normalization)
|
||||||
|
|
||||||
**Definition of Done:**
|
**Definition of Done:**
|
||||||
- [ ] English/German headers map correctly
|
- [ ] English/German headers map correctly
|
||||||
|
|
@ -385,7 +401,7 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c
|
||||||
|
|
||||||
**Tasks:**
|
**Tasks:**
|
||||||
- [ ] Implement `validate_row/3 (row_map, csv_line_number, opts)`
|
- [ ] Implement `validate_row/3 (row_map, csv_line_number, opts)`
|
||||||
- [ ] Required field presence (`first_name`, `last_name`, `email`)
|
- [ ] Required field presence (`email`)
|
||||||
- [ ] Email format validation (EctoCommons.EmailValidator)
|
- [ ] Email format validation (EctoCommons.EmailValidator)
|
||||||
- [ ] Trim values before validation
|
- [ ] Trim values before validation
|
||||||
- [ ] Gettext-backed error messages
|
- [ ] Gettext-backed error messages
|
||||||
|
|
@ -421,6 +437,10 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c
|
||||||
|
|
||||||
**Tasks:**
|
**Tasks:**
|
||||||
- [ ] Render import section only for admins
|
- [ ] Render import section only for admins
|
||||||
|
- [ ] **Add prominent UI notice about custom fields:**
|
||||||
|
- Display alert/info box: "Custom fields must be created in Mila before importing CSV files with custom field columns"
|
||||||
|
- Explain: "Use the custom field name as the CSV column header (same normalization as member fields applies)"
|
||||||
|
- Add link to custom fields management section
|
||||||
- [ ] Configure `allow_upload/3`:
|
- [ ] Configure `allow_upload/3`:
|
||||||
- `.csv` only, `max_entries: 1`, `max_file_size: 10MB`, `auto_upload: false`
|
- `.csv` only, `max_entries: 1`, `max_file_size: 10MB`, `auto_upload: false`
|
||||||
- [ ] `handle_event("start_import", ...)`:
|
- [ ] `handle_event("start_import", ...)`:
|
||||||
|
|
@ -439,6 +459,7 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c
|
||||||
- Success count
|
- Success count
|
||||||
- Failure count
|
- Failure count
|
||||||
- Error list (line number + message + field)
|
- Error list (line number + message + field)
|
||||||
|
- **Warning messages for unknown custom field columns** (non-existent names) shown in results
|
||||||
|
|
||||||
**Template links:**
|
**Template links:**
|
||||||
- Link `/templates/member_import_en.csv` and `/templates/member_import_de.csv` via Phoenix static path helpers.
|
- Link `/templates/member_import_en.csv` and `/templates/member_import_de.csv` via Phoenix static path helpers.
|
||||||
|
|
@ -471,8 +492,10 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c
|
||||||
|
|
||||||
**Tasks:**
|
**Tasks:**
|
||||||
- [ ] Fixtures:
|
- [ ] Fixtures:
|
||||||
- valid EN/DE
|
- valid EN/DE (core fields only)
|
||||||
|
- valid with custom fields
|
||||||
- invalid
|
- invalid
|
||||||
|
- unknown custom field name (non-existent, should show warning)
|
||||||
- too many rows (1,001)
|
- too many rows (1,001)
|
||||||
- BOM + `;` delimiter fixture
|
- BOM + `;` delimiter fixture
|
||||||
- fixture with empty line(s) to validate correct line numbers
|
- fixture with empty line(s) to validate correct line numbers
|
||||||
|
|
@ -481,6 +504,8 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c
|
||||||
- upload + start import
|
- upload + start import
|
||||||
- success + error rendering
|
- success + error rendering
|
||||||
- row limit + file size errors
|
- row limit + file size errors
|
||||||
|
- custom field import success
|
||||||
|
- custom field import warning (non-existent name, column ignored)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
@ -495,12 +520,42 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
### Issue #11: Custom Field Import (Optional - v1.1)
|
### Issue #11: Custom Field Import
|
||||||
|
|
||||||
**Dependencies:** Issue #10
|
**Dependencies:** Issue #6 (Persistence)
|
||||||
**Status:** Optional
|
|
||||||
|
|
||||||
*(unchanged — intentionally deferred)*
|
**Priority:** High (Core v1 Feature)
|
||||||
|
|
||||||
|
**Goal:** Support importing custom field values from CSV columns. Custom fields should exist in Mila before import for best results.
|
||||||
|
|
||||||
|
**Important Requirements:**
|
||||||
|
- **Custom fields should be created in Mila first** - Unknown custom field columns will be ignored with a warning message
|
||||||
|
- CSV headers for custom fields must match the custom field **name** exactly (same normalization as member fields applies)
|
||||||
|
- Custom field values are validated according to the custom field type (string, integer, boolean, date, email)
|
||||||
|
- Unknown custom field columns (non-existent names) will be ignored with a warning - import continues
|
||||||
|
|
||||||
|
**Tasks:**
|
||||||
|
- [ ] Extend `header_mapper.ex` to detect custom field columns by name (using same normalization as member fields)
|
||||||
|
- [ ] Query existing custom fields during `prepare/2` to map custom field columns
|
||||||
|
- [ ] Collect unknown custom field columns and add warning messages (don't fail import)
|
||||||
|
- [ ] Map custom field CSV values to `CustomFieldValue` creation in `process_chunk/3`
|
||||||
|
- [ ] Handle custom field type validation (string, integer, boolean, date, email)
|
||||||
|
- [ ] Create `CustomFieldValue` records linked to members during import
|
||||||
|
- [ ] Update error messages to include custom field validation errors
|
||||||
|
- [ ] Add UI help text explaining custom field requirements:
|
||||||
|
- "Custom fields must be created in Mila before importing"
|
||||||
|
- "Use the custom field name as the CSV column header (same normalization as member fields)"
|
||||||
|
- Link to custom fields management section
|
||||||
|
- [ ] Update CSV templates documentation to explain custom field columns
|
||||||
|
- [ ] Add tests for custom field import (valid, invalid name, type validation, warning for unknown)
|
||||||
|
|
||||||
|
**Definition of Done:**
|
||||||
|
- [ ] Custom field columns are recognized by name (with normalization)
|
||||||
|
- [ ] Warning messages shown for unknown custom field columns (import continues)
|
||||||
|
- [ ] Custom field values are created and linked to members
|
||||||
|
- [ ] Type validation works for all custom field types
|
||||||
|
- [ ] UI clearly explains custom field requirements
|
||||||
|
- [ ] Tests cover custom field import scenarios (including warning for unknown names)
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
|
||||||
Loading…
Add table
Add a link
Reference in a new issue