132 lines
12 KiB
Markdown
132 lines
12 KiB
Markdown
# Development Progress Log
|
||
|
||
**Project:** Mila — Membership Management System (AGPLv3)
|
||
**Status:** Early Development (⚠️ Not Production Ready)
|
||
|
||
A coarse history of what was built and, more importantly, *why*. Per-feature status lives in [`feature-roadmap.md`](feature-roadmap.md); setup/workflow/conventions live in [`README.md`](../README.md) and [`CODE_GUIDELINES.md`](../CODE_GUIDELINES.md). This file keeps the chronological narrative, the architecture decisions and deviations, the migration index, and the hard-won gotchas.
|
||
|
||
---
|
||
|
||
## Chronological narrative
|
||
|
||
Sprint dates and per-feature ✅/open/missing status are in [`feature-roadmap.md`](feature-roadmap.md); this is the coarse arc.
|
||
|
||
- **Sprint 0–2 (foundation):** `mix phx.new mv --no-ecto --no-mailer` (Ash instead of Ecto; Swoosh added later). Stack pinned via asdf (`.tool-versions`): Elixir 1.18.3-otp-27, Erlang 27.3.4, Just 1.46.0. PostgreSQL extensions (see `Mv.Repo.installed_extensions/0`): `ash-functions`, `citext`, `pg_trgm`. UUIDv7 comes from a custom `uuid_generate_v7()` SQL function defined in the extensions migration, not from an extension.
|
||
- **Sprint 3–5 (core):** Member CRUD, EAV custom-field system, Tailwind/DaisyUI UI, custom login + members landing page, seed data.
|
||
- **Sprint 6 (search):** Full-text search — weighted tsvector (names A, email/notes B, contact C), GIN index, auto-updating trigger, simple lexer (no German stemming yet).
|
||
- **Sprint 7 (sorting & links):** Sortable table headers (ARIA); optional 1:1 User↔Member link (User `belongs_to` Member, Member `has_one` User) as foundation for email sync.
|
||
- **Sprint 8 (email sync):** Bidirectional User↔Member email sync; User.email is source of truth on linking. Rules + decision tree in [`email-sync.md`](email-sync.md).
|
||
- **Sprint 9 (search + OIDC):** Fuzzy search (pg_trgm, 6 trigram GIN indexes, combined FTS+trigram, threshold 0.2, `word_similarity`). Secure OIDC account linking with password verification — see [`oidc-account-linking.md`](oidc-account-linking.md). Documentation suite + 100% `@moduledoc` coverage with Credo `ModuleDoc` enforcement.
|
||
- **Late 2025 → 2026:** Membership-fee system (types, calendar cycles, settings, status tracking); custom-field search + required-field validation; field-visibility settings; bulk email copy; sidebar navigation (DaisyUI drawer, WCAG 2.1 AA); CSV import (chunked, configurable limits, custom fields); groups (many-to-many, search-integrated); RBAC (4 permission sets, database-backed roles, resource policies, page-permission plug, system-actor pattern); onboarding/join requests (issue #308, TDD); statistics page MVP; SMTP configuration. See [`feature-roadmap.md`](feature-roadmap.md) and the dedicated docs per area.
|
||
|
||
---
|
||
|
||
## Architecture Decisions & Deviations
|
||
|
||
### Ash Framework over Ecto contexts
|
||
Declarative resources, built-in policy authorization, calculations/aggregates, migration code-gen. Trade-off: steeper learning curve, smaller ecosystem, more opinionated. Original plan was traditional Phoenix + Ecto; switched because Ash delivers more (policies, admin, codegen) with less code. Detailed structure in `CODE_GUIDELINES.md`.
|
||
|
||
### Domain-Driven structure
|
||
Organized by business domain (Accounts, Membership, MembershipFees) rather than technical layer, to keep business logic out of the web layer and scale to future domains.
|
||
|
||
### UUIDv7 for Members only
|
||
Members use `uuid_v7_primary_key :id` (sortable by creation time → better index locality, chronological ordering); Users and other resources keep standard `uuid_primary_key :id` (v4). The split is deliberate.
|
||
|
||
### No default `:create` action on User
|
||
`User` defines `defaults [:read]`, declares `:destroy` as a separate explicit named action, and exposes explicit creates (`create_user`, `register_with_password`, `register_with_oidc`) — **never** a default `:create`. Rationale: a default create would bypass the email-sync changes, so it is excluded as a safety measure.
|
||
|
||
### Bidirectional email sync — conditional, not always-on
|
||
Users and Members can exist independently; emails must stay in sync only when linked. Rejected alternatives: single shared email table (too restrictive — members without users still need emails); always-sync (needless work for unlinked entities); manual sync (error-prone). Chosen: conditional sync via custom Ash changes + validations — flexible, safe, performant. Source of truth is User.email on linking. Full rules + decision tree in [`email-sync.md`](email-sync.md).
|
||
|
||
### Custom fields — EAV with union types
|
||
`CustomField` defines the schema (name, `value_type` ∈ {string, integer, boolean, date, email}, required, show_in_overview); `CustomFieldValue` stores the polymorphic value via Ash `:union` type and `belongs_to :member` + `belongs_to :custom_field`. Chosen so clubs can add fields without schema migrations, with type safety. Constraints: one value per (member, custom_field) (composite unique index); values CASCADE-deleted with the member; custom-field types RESTRICT-protected while in use.
|
||
|
||
### Search — native PostgreSQL, not Elasticsearch/Meilisearch
|
||
Two tiers: weighted tsvector full-text (auto-updating trigger) + pg_trgm fuzzy (6 trigram GIN indexes, `similarity`/`word_similarity`, configurable threshold 0.2). Rejected Elasticsearch/Meilisearch: overkill for small/mid clubs, extra infra, and PostgreSQL FTS+fuzzy is sufficient for 10k+ members with better stack integration. German stemming remains a known gap (simple lexer).
|
||
|
||
### Authentication — multi-strategy
|
||
Password (bcrypt) + OIDC (Rauthy, self-hostable). `store_all_tokens? true`, JWT sessions, token revocation. Clubs pick the method; password is the non-SSO fallback.
|
||
|
||
### Deployment
|
||
Multi-stage Docker (Debian builder → slim runtime), assets via `mix assets.deploy`, Mix release, migrations via `Mv.Release.migrate`. Bandit over Cowboy (LiveView performance). Renovate runs the first week of each month (grouped mix/asdf/postgres); Elixir/Erlang auto-updates are **disabled** — OTP coupling and dependency compatibility require manual asdf control.
|
||
|
||
---
|
||
|
||
## Migration Index (26, chronological)
|
||
|
||
Timestamp → intent. Ash generates migrations from resource snapshots, so order matters and they must not be skipped.
|
||
|
||
1. `20250421101957_initialize_extensions_1` — PostgreSQL extensions (ash-functions, citext, pg_trgm) plus the custom `uuid_generate_v7()` SQL function (source of UUIDv7; not an extension)
|
||
2. `20250528163901_initial_migration` — core tables (members, custom_field_values, custom_fields — originally property_types/properties)
|
||
3. `20250617090641_member_fields` — member attributes expansion
|
||
4. `20250617132424_member_delete` — member deletion constraints
|
||
5. `20250620110849_add_accounts_domain_extensions` — accounts domain extensions
|
||
6. `20250620110850_add_accounts_domain` — users & tokens tables
|
||
7. `20250912085235_AddSearchVectorToMembers` — full-text search (tsvector + GIN index)
|
||
8. `20250926164519_member_relation` — User–Member link (optional 1:1)
|
||
9. `20250926180341_add_unique_email_to_members` — unique email constraint on members
|
||
10. `20251001141005_add_trigram_to_members` — fuzzy search (pg_trgm + 6 GIN trigram indexes)
|
||
11. `20251016130855_add_constraints_for_user_member_and_property` — email-sync constraints
|
||
12. `20251113163600_rename_properties_to_custom_fields_extensions_1` — rename properties extensions
|
||
13. `20251113163602_rename_properties_to_custom_fields` — rename property_types → custom_fields, properties → custom_field_values
|
||
14. `20251113180429_add_slug_to_custom_fields` — add slug to custom fields
|
||
15. `20251113183538_change_custom_field_delete_cascade` — change delete cascade behavior
|
||
16. `20251119160509_add_show_in_overview_to_custom_fields` — add show_in_overview flag
|
||
17. `20251127134451_add_settings_table` — create settings table (singleton)
|
||
18. `20251201115939_add_member_field_visibility_to_settings` — add member_field_visibility JSONB to settings
|
||
19. `20251202145404_remove_birth_date_from_members` — remove birth_date field
|
||
20. `20251204123714_add_custom_field_values_to_search_vector` — include custom field values in search vector
|
||
21. `20251211151449_add_membership_fees_tables` — create membership_fee_types and membership_fee_cycles tables
|
||
22. `20251211172549_remove_immutable_from_custom_fields` — remove immutable flag from custom fields
|
||
23. `20251211195058_add_membership_fee_settings` — add membership fee settings to settings table
|
||
24. `20251218113900_remove_paid_from_members` — remove paid boolean from members (replaced by cycle status)
|
||
25. `20260102155350_remove_phone_number_and_make_fields_optional` — remove phone_number; make first_name/last_name optional
|
||
26. `20260106161215_add_authorization_domain` — create roles table and add role_id to users
|
||
|
||
(Later migrations exist for join requests `20260309141437_add_join_requests`, join-form settings `20260310114701`, and groups `20260127141620_add_groups_and_member_groups` — added after this index was first cut; extend as needed.)
|
||
|
||
---
|
||
|
||
## Gotchas & Hard-won Lessons
|
||
|
||
### Ash `manage_relationship` validation (drove a real bug fix)
|
||
During validation (while `manage_relationship` is processing), the linked record lives in **`changeset.relationships`**, not `changeset.attributes` — `member_id` is still `nil` until the action completes:
|
||
|
||
```elixir
|
||
# during validation:
|
||
changeset.relationships.member = [{[%{id: "uuid"}], opts}]
|
||
changeset.attributes.member_id = nil # still nil!
|
||
# after action:
|
||
changeset.attributes.member_id = "uuid"
|
||
```
|
||
|
||
So a validation needing the linked id must read both sources:
|
||
|
||
```elixir
|
||
defp get_member_id_from_changeset(changeset) do
|
||
case Map.get(changeset.relationships, :member) do
|
||
[{[%{id: id}], _opts}] -> id # new link
|
||
_ -> Ash.Changeset.get_attribute(changeset, :member_id) # existing
|
||
end
|
||
end
|
||
```
|
||
|
||
This fixed email-validation false positives when linking a user and member that share an email.
|
||
|
||
### test_helper.exs clears Vereinfacht/OIDC ENV at startup
|
||
`test/test_helper.exs` clears Vereinfacht- and OIDC-related environment variables on boot. `Mv.Config` prefers ENV over database settings; without clearing, a developer's loaded `.env` would make OIDC sign-in redirect tests depend on the shell and go flaky (and risk hitting real APIs). Tests that need specific OIDC ENV set them in `setup` and restore via `on_exit`.
|
||
|
||
### LiveView + JavaScript: when to reach for JS
|
||
`push_event/3` → `window.addEventListener("phx:...")` is the idiomatic escape hatch. Use JS only for: direct DOM manipulation (autocomplete input values — LiveView DOM patching races on rapid state changes), browser APIs (clipboard), third-party libs, and preventing browser defaults (form submit on Enter). Do **not** use JS for form submissions, show/hide, server data fetching, or keyboard-navigation logic. Keyboard nav was implemented server-side (`phx-window-keydown`, ~45 lines) with a minimal hook (~13 lines) only to `preventDefault()` on Enter — ~20–50 ms roundtrip is imperceptible (perception threshold ~100 ms), versus ~80 lines for a full client-side solution. Don't prematurely optimize for latency.
|
||
|
||
### Common gotchas
|
||
1. **Ash actions are mandatory** — never use Ecto directly on Ash resources; always `Ash.create`/`Ash.update`/etc.
|
||
2. **Email sync only for linked entities** — cross-table email validation kicks in only on linking; unlinked users/members are not checked.
|
||
3. **Migrations run in order** — they depend on resource snapshots; don't skip.
|
||
4. **LiveView assigns are immutable** — return `{:noreply, assign(socket, ...)}`; never mutate `socket.assigns.x`.
|
||
5. **Reset test DB after schema changes** — `MIX_ENV=test mix ash.reset`.
|
||
6. **Docker networks** — dev uses `network_mode: host` for Rauthy; prod should use proper Docker networks.
|
||
7. **Secrets in `runtime.exs`, not `config.exs`** — `config.exs` is compile-time; `runtime.exs` reads env vars at runtime.
|
||
|
||
### Bulk email copy (#230)
|
||
Format `First Last <email>` with semicolon separator (compatible with all major email clients). Button shows the count of *visible* selected members (respects search/filter), not total selected. Server formats via `push_event/3`; client handles clipboard with an API + older-browser fallback.
|