docs: update implementation plan
This commit is contained in:
parent
b70dd3d98f
commit
bf93b4aa42
1 changed files with 110 additions and 60 deletions
|
|
@ -2,10 +2,29 @@
|
|||
|
||||
**Version:** 1.0
|
||||
**Date:** 2025-01-XX
|
||||
**Status:** Ready for Implementation
|
||||
**Status:** In Progress (Backend Complete, UI Pending)
|
||||
**Related Documents:**
|
||||
- [Feature Roadmap](./feature-roadmap.md) - Overall feature planning
|
||||
|
||||
## Implementation Status
|
||||
|
||||
**Completed Issues:**
|
||||
- ✅ Issue #1: CSV Specification & Static Template Files
|
||||
- ✅ Issue #2: Import Service Module Skeleton
|
||||
- ✅ Issue #3: CSV Parsing + Delimiter Auto-Detection + BOM Handling
|
||||
- ✅ Issue #4: Header Normalization + Per-Header Mapping
|
||||
- ✅ Issue #5: Validation (Required Fields) + Error Formatting
|
||||
- ✅ Issue #6: Persistence via Ash Create + Per-Row Error Capture (with Error-Capping)
|
||||
- ✅ Issue #11: Custom Field Import (Backend)
|
||||
|
||||
**In Progress / Pending:**
|
||||
- ⏳ Issue #7: Admin Global Settings LiveView UI (Upload + Start Import + Results)
|
||||
- ⏳ Issue #8: Authorization + Limits
|
||||
- ⏳ Issue #9: End-to-End LiveView Tests + Fixtures
|
||||
- ⏳ Issue #10: Documentation Polish
|
||||
|
||||
**Latest Update:** Error-Capping in `process_chunk/4` implemented (2025-01-XX)
|
||||
|
||||
---
|
||||
|
||||
## Table of Contents
|
||||
|
|
@ -332,19 +351,24 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c
|
|||
|
||||
**Dependencies:** None
|
||||
|
||||
**Status:** ✅ **COMPLETED**
|
||||
|
||||
**Goal:** Define CSV contract and add static templates.
|
||||
|
||||
**Tasks:**
|
||||
- [ ] Finalize header mapping variants
|
||||
- [ ] Document normalization rules
|
||||
- [ ] Document delimiter detection strategy
|
||||
- [ ] Create templates in `priv/static/templates/` (UTF-8 with BOM)
|
||||
- [ ] Document template URLs and how to link them from LiveView
|
||||
- [ ] Document line number semantics (physical CSV line numbers)
|
||||
- [x] Finalize header mapping variants
|
||||
- [x] Document normalization rules
|
||||
- [x] Document delimiter detection strategy
|
||||
- [x] Create templates in `priv/static/templates/` (UTF-8 with BOM)
|
||||
- `member_import_en.csv` with English headers
|
||||
- `member_import_de.csv` with German headers
|
||||
- [x] Document template URLs and how to link them from LiveView
|
||||
- [x] Document line number semantics (physical CSV line numbers)
|
||||
- [x] Templates included in `MvWeb.static_paths()` configuration
|
||||
|
||||
**Definition of Done:**
|
||||
- [ ] Templates open cleanly in Excel/LibreOffice
|
||||
- [ ] CSV spec section complete
|
||||
- [x] Templates open cleanly in Excel/LibreOffice
|
||||
- [x] CSV spec section complete
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -352,18 +376,20 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c
|
|||
|
||||
**Dependencies:** None
|
||||
|
||||
**Status:** ✅ **COMPLETED**
|
||||
|
||||
**Goal:** Create service API and error types.
|
||||
|
||||
**API (recommended):**
|
||||
- `prepare/2` — parse + map + limit checks, returns import_state
|
||||
- `process_chunk/3` — process one chunk (pure-ish), returns per-chunk results
|
||||
- `process_chunk/4` — process one chunk (pure-ish), returns per-chunk results
|
||||
|
||||
**Tasks:**
|
||||
- [ ] Create `lib/mv/membership/import/member_csv.ex`
|
||||
- [ ] Define public function: `prepare/2 (file_content, opts \\ [])`
|
||||
- [ ] Define public function: `process_chunk/3 (chunk_rows_with_lines, column_map, opts \\ [])`
|
||||
- [ ] Define error struct: `%MemberCSV.Error{csv_line_number: integer, field: atom | nil, message: String.t}`
|
||||
- [ ] Document module + API
|
||||
- [x] Create `lib/mv/membership/import/member_csv.ex`
|
||||
- [x] Define public function: `prepare/2 (file_content, opts \\ [])`
|
||||
- [x] Define public function: `process_chunk/4 (chunk_rows_with_lines, column_map, custom_field_map, opts \\ [])`
|
||||
- [x] Define error struct: `%MemberCSV.Error{csv_line_number: integer, field: atom | nil, message: String.t}`
|
||||
- [x] Document module + API
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -371,24 +397,26 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c
|
|||
|
||||
**Dependencies:** Issue #2
|
||||
|
||||
**Status:** ✅ **COMPLETED**
|
||||
|
||||
**Goal:** Parse CSV robustly with correct delimiter detection and BOM handling.
|
||||
|
||||
**Tasks:**
|
||||
- [ ] Verify/add NimbleCSV dependency (`{:nimble_csv, "~> 1.0"}`)
|
||||
- [ ] Create `lib/mv/membership/import/csv_parser.ex`
|
||||
- [ ] Implement `strip_bom/1` and apply it **before** any header handling
|
||||
- [ ] Handle `\r\n` and `\n` line endings (trim `\r` on header record)
|
||||
- [ ] Detect delimiter via header recognition (try `;` and `,`)
|
||||
- [ ] Parse CSV and return:
|
||||
- [x] Verify/add NimbleCSV dependency (`{:nimble_csv, "~> 1.0"}`)
|
||||
- [x] Create `lib/mv/membership/import/csv_parser.ex`
|
||||
- [x] Implement `strip_bom/1` and apply it **before** any header handling
|
||||
- [x] Handle `\r\n` and `\n` line endings (trim `\r` on header record)
|
||||
- [x] Detect delimiter via header recognition (try `;` and `,`)
|
||||
- [x] Parse CSV and return:
|
||||
- `headers :: [String.t()]`
|
||||
- `rows :: [{csv_line_number, [String.t()]}]` or directly `[{csv_line_number, row_map}]`
|
||||
- [ ] Skip completely empty records (but preserve correct physical line numbers)
|
||||
- [ ] Return `{:ok, headers, rows}` or `{:error, reason}`
|
||||
- `rows :: [{csv_line_number, [String.t()]}]` with correct physical line numbers
|
||||
- [x] Skip completely empty records (but preserve correct physical line numbers)
|
||||
- [x] Return `{:ok, headers, rows}` or `{:error, reason}`
|
||||
|
||||
**Definition of Done:**
|
||||
- [ ] BOM handling works (Excel exports)
|
||||
- [ ] Delimiter detection works reliably
|
||||
- [ ] Rows carry correct `csv_line_number`
|
||||
- [x] BOM handling works (Excel exports)
|
||||
- [x] Delimiter detection works reliably
|
||||
- [x] Rows carry correct `csv_line_number`
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -396,20 +424,22 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c
|
|||
|
||||
**Dependencies:** Issue #3
|
||||
|
||||
**Status:** ✅ **COMPLETED**
|
||||
|
||||
**Goal:** Map each header individually to canonical fields (normalized comparison).
|
||||
|
||||
**Tasks:**
|
||||
- [ ] Create `lib/mv/membership/import/header_mapper.ex`
|
||||
- [ ] Implement `normalize_header/1`
|
||||
- [ ] Normalize mapping variants once and compare normalized strings
|
||||
- [ ] Build `column_map` (canonical field -> column index)
|
||||
- [ ] **Early abort if required headers missing** (`email`)
|
||||
- [ ] Ignore unknown columns (member fields only)
|
||||
- [ ] **Separate custom field column detection** (by name, with normalization)
|
||||
- [x] Create `lib/mv/membership/import/header_mapper.ex`
|
||||
- [x] Implement `normalize_header/1`
|
||||
- [x] Normalize mapping variants once and compare normalized strings
|
||||
- [x] Build `column_map` (canonical field -> column index)
|
||||
- [x] **Early abort if required headers missing** (`email`)
|
||||
- [x] Ignore unknown columns (member fields only)
|
||||
- [x] **Separate custom field column detection** (by name, with normalization)
|
||||
|
||||
**Definition of Done:**
|
||||
- [ ] English/German headers map correctly
|
||||
- [ ] Missing required columns fails fast
|
||||
- [x] English/German headers map correctly
|
||||
- [x] Missing required columns fails fast
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -417,14 +447,16 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c
|
|||
|
||||
**Dependencies:** Issue #4
|
||||
|
||||
**Status:** ✅ **COMPLETED**
|
||||
|
||||
**Goal:** Validate each row and return structured, translatable errors.
|
||||
|
||||
**Tasks:**
|
||||
- [ ] Implement `validate_row/3 (row_map, csv_line_number, opts)`
|
||||
- [ ] Required field presence (`email`)
|
||||
- [ ] Email format validation (EctoCommons.EmailValidator)
|
||||
- [ ] Trim values before validation
|
||||
- [ ] Gettext-backed error messages
|
||||
- [x] Implement `validate_row/3 (row_map, csv_line_number, opts)`
|
||||
- [x] Required field presence (`email`)
|
||||
- [x] Email format validation (EctoCommons.EmailValidator)
|
||||
- [x] Trim values before validation
|
||||
- [x] Gettext-backed error messages
|
||||
|
||||
---
|
||||
|
||||
|
|
@ -432,21 +464,32 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c
|
|||
|
||||
**Dependencies:** Issue #5
|
||||
|
||||
**Status:** ✅ **COMPLETED**
|
||||
|
||||
**Goal:** Create members and capture errors per row with correct CSV line numbers.
|
||||
|
||||
**Tasks:**
|
||||
- [ ] Implement `process_chunk/3` in service:
|
||||
- [x] Implement `process_chunk/4` in service:
|
||||
- Input: `[{csv_line_number, row_map}]`
|
||||
- Validate + create sequentially
|
||||
- Collect counts + first 50 errors (per import overall; LiveView enforces cap across chunks)
|
||||
- [ ] Implement Ash error formatter helper:
|
||||
- **Error-Capping:** Supports `existing_error_count` and `max_errors` in opts (default: 50)
|
||||
- **Error-Capping:** Only collects errors if under limit, but continues processing all rows
|
||||
- **Error-Capping:** `failed` count is always accurate, even when errors are capped
|
||||
- [x] Implement Ash error formatter helper:
|
||||
- Convert `Ash.Error.Invalid` into `%MemberCSV.Error{}`
|
||||
- Prefer field-level errors where possible (attach `field` atom)
|
||||
- Handle unique email constraint error as user-friendly message
|
||||
- [ ] Map row_map to Ash attrs (`%{first_name: ..., ...}`)
|
||||
- [x] Map row_map to Ash attrs (`%{first_name: ..., ...}`)
|
||||
- [x] Custom field value processing and creation
|
||||
|
||||
**Important:** **Do not recompute line numbers** in this layer—use the ones provided by the parser.
|
||||
|
||||
**Implementation Notes:**
|
||||
- `process_chunk/4` accepts `opts` with `existing_error_count` and `max_errors` for error capping across chunks
|
||||
- Error capping respects the limit per import overall (not per chunk)
|
||||
- Processing continues even after error limit is reached (for accurate counts)
|
||||
|
||||
---
|
||||
|
||||
### Issue #7: Admin Global Settings LiveView UI (Upload + Start Import + Results + Template Links)
|
||||
|
|
@ -546,6 +589,8 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c
|
|||
|
||||
**Priority:** High (Core v1 Feature)
|
||||
|
||||
**Status:** ✅ **COMPLETED** (Backend Implementation)
|
||||
|
||||
**Goal:** Support importing custom field values from CSV columns. Custom fields should exist in Mila before import for best results.
|
||||
|
||||
**Important Requirements:**
|
||||
|
|
@ -555,27 +600,32 @@ Use `Mv.Authorization.PermissionSets` (preferred) instead of hard-coded string c
|
|||
- Unknown custom field columns (non-existent names) will be ignored with a warning - import continues
|
||||
|
||||
**Tasks:**
|
||||
- [ ] Extend `header_mapper.ex` to detect custom field columns by name (using same normalization as member fields)
|
||||
- [ ] Query existing custom fields during `prepare/2` to map custom field columns
|
||||
- [ ] Collect unknown custom field columns and add warning messages (don't fail import)
|
||||
- [ ] Map custom field CSV values to `CustomFieldValue` creation in `process_chunk/3`
|
||||
- [ ] Handle custom field type validation (string, integer, boolean, date, email)
|
||||
- [ ] Create `CustomFieldValue` records linked to members during import
|
||||
- [ ] Update error messages to include custom field validation errors
|
||||
- [ ] Add UI help text explaining custom field requirements:
|
||||
- [x] Extend `header_mapper.ex` to detect custom field columns by name (using same normalization as member fields)
|
||||
- [x] Query existing custom fields during `prepare/2` to map custom field columns
|
||||
- [x] Collect unknown custom field columns and add warning messages (don't fail import)
|
||||
- [x] Map custom field CSV values to `CustomFieldValue` creation in `process_chunk/4`
|
||||
- [x] Handle custom field type validation (string, integer, boolean, date, email)
|
||||
- [x] Create `CustomFieldValue` records linked to members during import
|
||||
- [ ] Update error messages to include custom field validation errors (if needed)
|
||||
- [ ] Add UI help text explaining custom field requirements (pending Issue #7):
|
||||
- "Custom fields must be created in Mila before importing"
|
||||
- "Use the custom field name as the CSV column header (same normalization as member fields)"
|
||||
- Link to custom fields management section
|
||||
- [ ] Update CSV templates documentation to explain custom field columns
|
||||
- [ ] Add tests for custom field import (valid, invalid name, type validation, warning for unknown)
|
||||
- [ ] Update CSV templates documentation to explain custom field columns (pending Issue #1)
|
||||
- [x] Add tests for custom field import (valid, invalid name, type validation, warning for unknown)
|
||||
|
||||
**Definition of Done:**
|
||||
- [ ] Custom field columns are recognized by name (with normalization)
|
||||
- [ ] Warning messages shown for unknown custom field columns (import continues)
|
||||
- [ ] Custom field values are created and linked to members
|
||||
- [ ] Type validation works for all custom field types
|
||||
- [ ] UI clearly explains custom field requirements
|
||||
- [ ] Tests cover custom field import scenarios (including warning for unknown names)
|
||||
- [x] Custom field columns are recognized by name (with normalization)
|
||||
- [x] Warning messages shown for unknown custom field columns (import continues)
|
||||
- [x] Custom field values are created and linked to members
|
||||
- [x] Type validation works for all custom field types
|
||||
- [ ] UI clearly explains custom field requirements (pending Issue #7)
|
||||
- [x] Tests cover custom field import scenarios (including warning for unknown names)
|
||||
|
||||
**Implementation Notes:**
|
||||
- Custom field lookup is built in `prepare/2` and passed via `custom_field_lookup` in opts
|
||||
- Custom field values are formatted according to type in `format_custom_field_value/2`
|
||||
- Unknown custom field columns generate warnings in `import_state.warnings`
|
||||
|
||||
---
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue