diff --git a/docs/custom-fields-search-performance.md b/docs/custom-fields-search-performance.md deleted file mode 100644 index 3987c85..0000000 --- a/docs/custom-fields-search-performance.md +++ /dev/null @@ -1,243 +0,0 @@ -# Performance Analysis: Custom Fields in Search Vector - -## Current Implementation - -The search vector includes custom field values via database triggers that: -1. Aggregate all custom field values for a member -2. Extract values from JSONB format -3. Add them to the search_vector with weight 'C' - -## Performance Considerations - -### 1. Trigger Performance on Member Updates - -**Current Implementation:** -- `members_search_vector_trigger()` executes a subquery on every INSERT/UPDATE: - ```sql - SELECT string_agg(...) FROM custom_field_values WHERE member_id = NEW.id - ``` - -**Performance Impact:** -- ✅ **Good:** Index on `member_id` exists (`custom_field_values_member_id_idx`) -- ✅ **Good:** Subquery only runs for the affected member -- ⚠️ **Potential Issue:** With many custom fields per member (e.g., 50+), aggregation could be slower -- ⚠️ **Potential Issue:** JSONB extraction (`value->>'_union_value'`) is relatively fast but adds overhead - -**Expected Performance:** -- **Small scale (< 10 custom fields per member):** Negligible impact (< 5ms per operation) -- **Medium scale (10-30 custom fields):** Minor impact (5-20ms per operation) -- **Large scale (30+ custom fields):** Noticeable impact (20-50ms+ per operation) - -### 2. Trigger Performance on Custom Field Value Changes - -**Current Implementation:** -- `update_member_search_vector_from_custom_field_value()` executes on every INSERT/UPDATE/DELETE on `custom_field_values` -- **Optimized:** Only fetches required member fields (not full record) to reduce overhead -- **Optimized:** Skips re-aggregation on UPDATE if value hasn't actually changed -- Aggregates all custom field values, then updates member search_vector - -**Performance Impact:** -- ✅ **Good:** Index on `member_id` ensures fast lookup -- ✅ **Optimized:** Only required fields are fetched (first_name, last_name, email, etc.) instead of full record -- ✅ **Optimized:** UPDATE operations that don't change the value skip expensive re-aggregation (early return) -- ⚠️ **Note:** Re-aggregation is still necessary when values change (required for search_vector consistency) -- ⚠️ **Critical:** Bulk operations (e.g., importing 1000 members with custom fields) will trigger this for each row - -**Expected Performance:** -- **Single operation (value changed):** 3-10ms per custom field value change (improved from 5-15ms) -- **Single operation (value unchanged):** <1ms (early return, no aggregation) -- **Bulk operations:** Could be slow (consider disabling trigger temporarily) - -### 3. Search Vector Size - -**Current Constraints:** -- String values: max 10,000 characters per custom field -- No limit on number of custom fields per member -- tsvector has no explicit size limit, but very large vectors can cause issues - -**Potential Issues:** -- **Theoretical maximum:** If a member has 100 custom fields with 10,000 char strings each, the aggregated text could be ~1MB -- **Practical concern:** Very large search vectors (> 100KB) can slow down: - - Index updates (GIN index maintenance) - - Search queries (tsvector operations) - - Trigger execution time - -**Recommendation:** -- Monitor search_vector size in production -- Consider limiting total custom field content per member if needed -- PostgreSQL can handle large tsvectors, but performance degrades gradually - -### 4. Initial Migration Performance - -**Current Implementation:** -- Updates ALL members in a single transaction: - ```sql - UPDATE members m SET search_vector = ... (subquery for each member) - ``` - -**Performance Impact:** -- ⚠️ **Potential Issue:** With 10,000+ members, this could take minutes -- ⚠️ **Potential Issue:** Single transaction locks the members table -- ⚠️ **Potential Issue:** If migration fails, entire rollback required - -**Recommendation:** -- For large datasets (> 10,000 members), consider: - - Batch updates (e.g., 1000 members at a time) - - Run during maintenance window - - Monitor progress - -### 5. Search Query Performance - -**Current Implementation:** -- Full-text search uses GIN index on `search_vector` (fast) -- Additional LIKE queries on `custom_field_values` for substring matching: - ```sql - EXISTS (SELECT 1 FROM custom_field_values WHERE member_id = id AND ... LIKE ...) - ``` - -**Performance Impact:** -- ✅ **Good:** GIN index on `search_vector` is very fast -- ⚠️ **Potential Issue:** LIKE queries on JSONB are not indexed (sequential scan) -- ⚠️ **Potential Issue:** EXISTS subquery runs for every search, even if search_vector match is found -- ⚠️ **Potential Issue:** With many custom fields, the LIKE queries could be slow - -**Expected Performance:** -- **With GIN index match:** Very fast (< 10ms for typical queries) -- **Without GIN index match (fallback to LIKE):** Slower (10-100ms depending on data size) -- **Worst case:** Sequential scan of all custom_field_values for all members - -## Recommendations - -### Short-term (Current Implementation) - -1. **Monitor Performance:** - - Add logging for trigger execution time - - Monitor search_vector size distribution - - Track search query performance - -2. **Index Verification:** - - Ensure `custom_field_values_member_id_idx` exists and is used - - Verify GIN index on `search_vector` is maintained - -3. **Bulk Operations:** - - For bulk imports, consider temporarily disabling the custom_field_values trigger - - Re-enable and update search_vectors in batch after import - -### Medium-term Optimizations - -1. **✅ Optimize Trigger Function (FULLY IMPLEMENTED):** - - ✅ Only fetch required member fields instead of full record (reduces overhead) - - ✅ Skip re-aggregation on UPDATE if value hasn't actually changed (early return optimization) - -2. **Limit Search Vector Size:** - - Truncate very long custom field values (e.g., first 1000 chars) - - Add warning if aggregated text exceeds threshold - -3. **Optimize LIKE Queries:** - - Consider adding a generated column for searchable text - - Or use a materialized view for custom field search - -### Long-term Considerations - -1. **Alternative Approaches:** - - Separate search index table for custom fields - - Use Elasticsearch or similar for advanced search - - Materialized view for search optimization - -2. **Scaling Strategy:** - - If performance becomes an issue with 100+ custom fields per member: - - Consider limiting which custom fields are searchable - - Use a separate search service - - Implement search result caching - -## Performance Benchmarks (Estimated) - -Based on typical PostgreSQL performance: - -| Scenario | Members | Custom Fields/Member | Expected Impact | -|----------|---------|---------------------|-----------------| -| Small | < 1,000 | < 10 | Negligible (< 5ms per operation) | -| Medium | 1,000-10,000 | 10-30 | Minor (5-20ms per operation) | -| Large | 10,000-100,000 | 30-50 | Noticeable (20-50ms per operation) | -| Very Large | > 100,000 | 50+ | Significant (50-200ms+ per operation) | - -## Monitoring Queries - -```sql --- Check search_vector size distribution -SELECT - pg_size_pretty(octet_length(search_vector::text)) as size, - COUNT(*) as member_count -FROM members -WHERE search_vector IS NOT NULL -GROUP BY octet_length(search_vector::text) -ORDER BY octet_length(search_vector::text) DESC -LIMIT 20; - --- Check average custom fields per member -SELECT - AVG(custom_field_count) as avg_custom_fields, - MAX(custom_field_count) as max_custom_fields -FROM ( - SELECT member_id, COUNT(*) as custom_field_count - FROM custom_field_values - GROUP BY member_id -) subq; - --- Check trigger execution time (requires pg_stat_statements) -SELECT - mean_exec_time, - calls, - query -FROM pg_stat_statements -WHERE query LIKE '%members_search_vector_trigger%' -ORDER BY mean_exec_time DESC; -``` - -## Code Quality Improvements (Post-Review) - -### Refactored Search Implementation - -The search query has been refactored for better maintainability and clarity: - -**Before:** Single large OR-chain with mixed search types (hard to maintain) - -**After:** Modular functions grouped by search type: -- `build_fts_filter/1` - Full-text search (highest priority, fastest) -- `build_substring_filter/2` - Substring matching on structured fields -- `build_custom_field_filter/1` - Custom field value search (JSONB LIKE) -- `build_fuzzy_filter/2` - Trigram/fuzzy matching for names and streets - -**Benefits:** -- ✅ Clear separation of concerns -- ✅ Easier to maintain and test -- ✅ Better documentation of search priority -- ✅ Easier to optimize individual search types - -**Search Priority Order:** -1. **FTS (Full-Text Search)** - Fastest, uses GIN index on search_vector -2. **Substring** - For structured fields (postal_code, phone_number, etc.) -3. **Custom Fields** - JSONB LIKE queries (fallback for substring matching) -4. **Fuzzy Matching** - Trigram similarity for names and streets - -## Conclusion - -The current implementation is **well-optimized for typical use cases** (< 30 custom fields per member, < 10,000 members). For larger scales, monitoring and potential optimizations may be needed. - -**Key Strengths:** -- Indexed lookups (member_id index) -- Efficient GIN index for search -- Trigger-based automatic updates -- Modular, maintainable search code structure - -**Key Weaknesses:** -- LIKE queries on JSONB (not indexed) -- Re-aggregation on every custom field change (necessary for consistency) -- Potential size issues with many/large custom fields -- Substring searches (contains/ILIKE) not index-optimized - -**Recent Optimizations:** -- ✅ Trigger function optimized to fetch only required fields (reduces overhead by ~30-50%) -- ✅ Early return on UPDATE when value hasn't changed (skips expensive re-aggregation, <1ms vs 3-10ms) -- ✅ Improved performance for custom field value updates (3-10ms vs 5-15ms when value changes) - diff --git a/docs/database-schema-readme.md b/docs/database-schema-readme.md index 6457db5..1644f2a 100644 --- a/docs/database-schema-readme.md +++ b/docs/database-schema-readme.md @@ -168,16 +168,9 @@ Member (1) → (N) Properties ### Weighted Fields - **Weight A (highest):** first_name, last_name - **Weight B:** email, notes -- **Weight C:** phone_number, city, street, house_number, postal_code, custom_field_values +- **Weight C:** phone_number, city, street, house_number, postal_code - **Weight D (lowest):** join_date, exit_date -### Custom Field Values in Search -Custom field values are automatically included in the search vector: -- All custom field values (string, integer, boolean, date, email) are aggregated and added to the search vector -- Values are converted to text format for indexing -- Custom field values receive weight 'C' (same as phone_number, city, etc.) -- The search vector is automatically updated when custom field values are created, updated, or deleted via database triggers - ### Usage Example ```sql SELECT * FROM members diff --git a/lib/membership/member.ex b/lib/membership/member.ex index d29a759..b788dc9 100644 --- a/lib/membership/member.ex +++ b/lib/membership/member.ex @@ -29,9 +29,7 @@ defmodule Mv.Membership.Member do ## Full-Text Search Members have a `search_vector` attribute (tsvector) that is automatically - updated via database trigger. Search includes name, email, notes, contact fields, - and all custom field values. Custom field values are automatically included in - the search vector with weight 'C' (same as phone_number, city, etc.). + updated via database trigger. Search includes name, email, notes, and contact fields. """ use Ash.Resource, domain: Mv.Membership, @@ -42,21 +40,6 @@ defmodule Mv.Membership.Member do # Module constants @member_search_limit 10 - - # Similarity threshold for fuzzy name/address matching. - # Lower value = more results but less accurate (0.1-0.9) - # - # Fuzzy matching uses two complementary strategies: - # 1. % operator: Fast GIN-index-based matching using server-wide threshold (default 0.3) - # - Catches exact trigram matches quickly via index - # 2. similarity/word_similarity functions: Precise matching with this configurable threshold - # - Catches partial matches that % operator might miss - # - # Value 0.2 chosen based on testing with typical German names: - # - "Müller" vs "Mueller": similarity ~0.65 ✓ - # - "Schmidt" vs "Schmitt": similarity ~0.75 ✓ - # - "Wagner" vs "Wegner": similarity ~0.55 ✓ - # - Random unrelated names: similarity ~0.15 ✗ @default_similarity_threshold 0.2 # Use constants from Mv.Constants for member fields @@ -156,21 +139,30 @@ defmodule Mv.Membership.Member do if is_binary(q) and String.trim(q) != "" do q2 = String.trim(q) - # Sanitize for LIKE patterns (escape % and _), limit length to 100 chars - q2_sanitized = sanitize_search_query(q2) - pat = "%" <> q2_sanitized <> "%" - - # Build search filters grouped by search type for maintainability - # Priority: FTS > Substring > Custom Fields > Fuzzy Matching - # Note: FTS and fuzzy use q2 (unsanitized), LIKE-based filters use pat (sanitized) - fts_match = build_fts_filter(q2) - substring_match = build_substring_filter(q2_sanitized, pat) - custom_field_match = build_custom_field_filter(pat) - fuzzy_match = build_fuzzy_filter(q2, threshold) + pat = "%" <> q2 <> "%" + # FTS as main filter and fuzzy search just for first name, last name and strees query |> Ash.Query.filter( - expr(^fts_match or ^substring_match or ^custom_field_match or ^fuzzy_match) + expr( + # Substring on numeric-like fields (best effort, supports middle substrings) + fragment("search_vector @@ websearch_to_tsquery('simple', ?)", ^q2) or + fragment("search_vector @@ plainto_tsquery('simple', ?)", ^q2) or + contains(postal_code, ^q2) or + contains(house_number, ^q2) or + contains(phone_number, ^q2) or + contains(email, ^q2) or + contains(city, ^q2) or ilike(city, ^pat) or + fragment("? % first_name", ^q2) or + fragment("? % last_name", ^q2) or + fragment("? % street", ^q2) or + fragment("word_similarity(?, first_name) > ?", ^q2, ^threshold) or + fragment("word_similarity(?, last_name) > ?", ^q2, ^threshold) or + fragment("word_similarity(?, street) > ?", ^q2, ^threshold) or + fragment("similarity(first_name, ?) > ?", ^q2, ^threshold) or + fragment("similarity(last_name, ?) > ?", ^q2, ^threshold) or + fragment("similarity(street, ?) > ?", ^q2, ^threshold) + ) ) else query @@ -484,6 +476,7 @@ defmodule Mv.Membership.Member do - `query` - Ash.Query.t() to apply search to - `opts` - Keyword list or map with search options: - `:query` or `"query"` - Search string + - `:fields` or `"fields"` - Optional field restrictions ## Returns - Modified Ash.Query.t() with search filters applied @@ -504,103 +497,16 @@ defmodule Mv.Membership.Member do if String.trim(q) == "" do query else - Ash.Query.for_read(query, :search, %{query: q}) + args = + case opts[:fields] || opts["fields"] do + nil -> %{query: q} + fields -> %{query: q, fields: fields} + end + + Ash.Query.for_read(query, :search, args) end end - # ============================================================================ - # Search Input Sanitization - # ============================================================================ - - # Sanitizes search input to prevent LIKE pattern injection. - # Escapes SQL LIKE wildcards (% and _) and limits query length. - # - # ## Examples - # - # iex> sanitize_search_query("test%injection") - # "test\\%injection" - # - # iex> sanitize_search_query("very_long_search") - # "very\\_long\\_search" - # - defp sanitize_search_query(query) when is_binary(query) do - query - |> String.slice(0, 100) - |> String.replace("\\", "\\\\") - |> String.replace("%", "\\%") - |> String.replace("_", "\\_") - end - - defp sanitize_search_query(_), do: "" - - # ============================================================================ - # Search Filter Builders - # ============================================================================ - # These functions build search filters grouped by search type for maintainability. - # Priority order: FTS > Substring > Custom Fields > Fuzzy Matching - - # Builds full-text search filter using tsvector (highest priority, fastest) - # Uses GIN index on search_vector for optimal performance - defp build_fts_filter(query) do - expr( - fragment("search_vector @@ websearch_to_tsquery('simple', ?)", ^query) or - fragment("search_vector @@ plainto_tsquery('simple', ?)", ^query) - ) - end - - # Builds substring search filter for structured fields - # Note: contains/2 uses ILIKE '%value%' which is not index-optimized - # Performance: Good for small datasets, may be slow on large tables - defp build_substring_filter(query, _pattern) do - expr( - contains(postal_code, ^query) or - contains(house_number, ^query) or - contains(phone_number, ^query) or - contains(email, ^query) or - contains(city, ^query) - ) - end - - # Builds search filter for custom field values using ILIKE on JSONB - # Note: ILIKE on JSONB is not index-optimized, may be slow with many custom fields - # This is a fallback for substring matching in custom fields (e.g., phone numbers) - # Uses ->> operator which always returns TEXT directly (no need for -> + ::text fallback) - # Important: `id` must be passed as parameter to correctly reference the outer members table - defp build_custom_field_filter(pattern) do - expr( - fragment( - "EXISTS (SELECT 1 FROM custom_field_values WHERE member_id = ? AND (value->>'_union_value' ILIKE ? OR value->>'value' ILIKE ?))", - id, - ^pattern, - ^pattern - ) - ) - end - - # Builds fuzzy/trigram matching filter for name and street fields. - # Uses pg_trgm extension with GIN indexes for performance. - # - # Two-tier matching strategy: - # - % operator: Uses server-wide pg_trgm.similarity_threshold (typically 0.3) - # for fast index-based initial filtering - # - similarity/word_similarity: Uses @default_similarity_threshold (0.2) - # for more lenient matching to catch edge cases - # - # Note: Requires trigram GIN indexes on first_name, last_name, street. - defp build_fuzzy_filter(query, threshold) do - expr( - fragment("? % first_name", ^query) or - fragment("? % last_name", ^query) or - fragment("? % street", ^query) or - fragment("word_similarity(?, first_name) > ?", ^query, ^threshold) or - fragment("word_similarity(?, last_name) > ?", ^query, ^threshold) or - fragment("word_similarity(?, street) > ?", ^query, ^threshold) or - fragment("similarity(first_name, ?) > ?", ^query, ^threshold) or - fragment("similarity(last_name, ?) > ?", ^query, ^threshold) or - fragment("similarity(street, ?) > ?", ^query, ^threshold) - ) - end - # Private helper to apply filters for :available_for_linking action # user_email: may be nil/empty when creating new user, or populated when editing # search_query: optional search term for fuzzy matching @@ -609,9 +515,9 @@ defmodule Mv.Membership.Member do # - Empty user_email ("") → email == "" is always false → only fuzzy search matches # - This allows a single filter expression instead of duplicating fuzzy search logic # - # Note: Custom field search is intentionally excluded from linking to optimize - # autocomplete performance. Custom fields are still searchable via the main - # member search which uses the indexed search_vector. + # Cyclomatic complexity is unavoidable here: PostgreSQL fuzzy search requires + # multiple OR conditions for good search quality (FTS + trigram similarity + substring) + # credo:disable-for-next-line Credo.Check.Refactor.CyclomaticComplexity defp apply_linking_filters(query, user_email, search_query) do has_search = search_query && String.trim(search_query) != "" # Use empty string instead of nil to simplify filter logic @@ -620,23 +526,35 @@ defmodule Mv.Membership.Member do if has_search do # Search query provided: return email-match OR fuzzy-search candidates trimmed_search = String.trim(search_query) - # Sanitize for LIKE patterns (contains uses ILIKE internally) - sanitized_search = sanitize_search_query(trimmed_search) - - # Build search filters - excluding custom_field_filter for performance - fts_match = build_fts_filter(trimmed_search) - fuzzy_match = build_fuzzy_filter(trimmed_search, @default_similarity_threshold) - email_substring_match = expr(contains(email, ^sanitized_search)) query |> Ash.Query.filter( expr( - # Email exact match has highest priority (for filter_by_email_match) - # If email is "", this is always false and search filters take over + # Email match candidate (for filter_by_email_match priority) + # If email is "", this is always false and fuzzy search takes over + # Fuzzy search candidates email == ^trimmed_email or - ^fts_match or - ^fuzzy_match or - ^email_substring_match + fragment("search_vector @@ websearch_to_tsquery('simple', ?)", ^trimmed_search) or + fragment("search_vector @@ plainto_tsquery('simple', ?)", ^trimmed_search) or + fragment("? % first_name", ^trimmed_search) or + fragment("? % last_name", ^trimmed_search) or + fragment("word_similarity(?, first_name) > 0.2", ^trimmed_search) or + fragment( + "word_similarity(?, last_name) > ?", + ^trimmed_search, + ^@default_similarity_threshold + ) or + fragment( + "similarity(first_name, ?) > ?", + ^trimmed_search, + ^@default_similarity_threshold + ) or + fragment( + "similarity(last_name, ?) > ?", + ^trimmed_search, + ^@default_similarity_threshold + ) or + contains(email, ^trimmed_search) ) ) else diff --git a/priv/repo/migrations/20251204123714_add_custom_field_values_to_search_vector.exs b/priv/repo/migrations/20251204123714_add_custom_field_values_to_search_vector.exs deleted file mode 100644 index 45a12e1..0000000 --- a/priv/repo/migrations/20251204123714_add_custom_field_values_to_search_vector.exs +++ /dev/null @@ -1,259 +0,0 @@ -defmodule Mv.Repo.Migrations.AddCustomFieldValuesToSearchVector do - @moduledoc """ - Extends the search_vector in members table to include custom_field_values. - - This migration: - 1. Updates the members_search_vector_trigger() function to include custom field values - 2. Creates a trigger function to update member search_vector when custom_field_values change - 3. Creates a trigger on custom_field_values table - 4. Updates existing search_vector values for all members - """ - - use Ecto.Migration - - def up do - # Update the main trigger function to include custom_field_values - execute(""" - CREATE OR REPLACE FUNCTION members_search_vector_trigger() RETURNS trigger AS $$ - DECLARE - custom_values_text text; - BEGIN - -- Aggregate all custom field values for this member - -- Support both formats: _union_type/_union_value (Ash format) and type/value (legacy) - -- ->> operator always returns TEXT directly (no need for -> + ::text fallback) - SELECT string_agg( - CASE - WHEN value ? '_union_value' THEN value->>'_union_value' - WHEN value ? 'value' THEN value->>'value' - ELSE '' - END, - ' ' - ) - INTO custom_values_text - FROM custom_field_values - WHERE member_id = NEW.id AND value IS NOT NULL; - - -- Build search_vector with member fields and custom field values - NEW.search_vector := - setweight(to_tsvector('simple', coalesce(NEW.first_name, '')), 'A') || - setweight(to_tsvector('simple', coalesce(NEW.last_name, '')), 'A') || - setweight(to_tsvector('simple', coalesce(NEW.email, '')), 'B') || - setweight(to_tsvector('simple', coalesce(NEW.phone_number, '')), 'C') || - setweight(to_tsvector('simple', coalesce(NEW.join_date::text, '')), 'D') || - setweight(to_tsvector('simple', coalesce(NEW.exit_date::text, '')), 'D') || - setweight(to_tsvector('simple', coalesce(NEW.notes, '')), 'B') || - setweight(to_tsvector('simple', coalesce(NEW.city, '')), 'C') || - setweight(to_tsvector('simple', coalesce(NEW.street, '')), 'C') || - setweight(to_tsvector('simple', coalesce(NEW.house_number::text, '')), 'C') || - setweight(to_tsvector('simple', coalesce(NEW.postal_code::text, '')), 'C') || - setweight(to_tsvector('simple', coalesce(custom_values_text, '')), 'C'); - RETURN NEW; - END - $$ LANGUAGE plpgsql; - """) - - # Create trigger function to update member search_vector when custom_field_values change - # Optimized: - # 1. Only fetch required fields instead of full member record to reduce overhead - # 2. Skip re-aggregation on UPDATE if value hasn't actually changed - execute(""" - CREATE FUNCTION update_member_search_vector_from_custom_field_value() RETURNS trigger AS $$ - DECLARE - member_id_val uuid; - member_first_name text; - member_last_name text; - member_email text; - member_phone_number text; - member_join_date date; - member_exit_date date; - member_notes text; - member_city text; - member_street text; - member_house_number text; - member_postal_code text; - custom_values_text text; - old_value_text text; - new_value_text text; - BEGIN - -- Get member ID from trigger context - member_id_val := COALESCE(NEW.member_id, OLD.member_id); - - -- Optimization: For UPDATE operations, check if value actually changed - -- If value hasn't changed, we can skip the expensive re-aggregation - IF TG_OP = 'UPDATE' THEN - -- Extract OLD value for comparison (handle both JSONB formats) - -- ->> operator always returns TEXT directly - old_value_text := COALESCE( - NULLIF(OLD.value->>'_union_value', ''), - NULLIF(OLD.value->>'value', ''), - '' - ); - - -- Extract NEW value for comparison (handle both JSONB formats) - new_value_text := COALESCE( - NULLIF(NEW.value->>'_union_value', ''), - NULLIF(NEW.value->>'value', ''), - '' - ); - - -- Check if value, member_id, or custom_field_id actually changed - -- If nothing changed, skip expensive re-aggregation - IF (old_value_text IS NOT DISTINCT FROM new_value_text) AND - (OLD.member_id IS NOT DISTINCT FROM NEW.member_id) AND - (OLD.custom_field_id IS NOT DISTINCT FROM NEW.custom_field_id) THEN - RETURN COALESCE(NEW, OLD); - END IF; - END IF; - - -- Fetch only required fields instead of full record (performance optimization) - SELECT - first_name, - last_name, - email, - phone_number, - join_date, - exit_date, - notes, - city, - street, - house_number, - postal_code - INTO - member_first_name, - member_last_name, - member_email, - member_phone_number, - member_join_date, - member_exit_date, - member_notes, - member_city, - member_street, - member_house_number, - member_postal_code - FROM members - WHERE id = member_id_val; - - -- Aggregate all custom field values for this member - -- Support both formats: _union_type/_union_value (Ash format) and type/value (legacy) - -- ->> operator always returns TEXT directly - SELECT string_agg( - CASE - WHEN value ? '_union_value' THEN value->>'_union_value' - WHEN value ? 'value' THEN value->>'value' - ELSE '' - END, - ' ' - ) - INTO custom_values_text - FROM custom_field_values - WHERE member_id = member_id_val AND value IS NOT NULL; - - -- Update the search_vector for the affected member - UPDATE members - SET search_vector = - setweight(to_tsvector('simple', coalesce(member_first_name, '')), 'A') || - setweight(to_tsvector('simple', coalesce(member_last_name, '')), 'A') || - setweight(to_tsvector('simple', coalesce(member_email, '')), 'B') || - setweight(to_tsvector('simple', coalesce(member_phone_number, '')), 'C') || - setweight(to_tsvector('simple', coalesce(member_join_date::text, '')), 'D') || - setweight(to_tsvector('simple', coalesce(member_exit_date::text, '')), 'D') || - setweight(to_tsvector('simple', coalesce(member_notes, '')), 'B') || - setweight(to_tsvector('simple', coalesce(member_city, '')), 'C') || - setweight(to_tsvector('simple', coalesce(member_street, '')), 'C') || - setweight(to_tsvector('simple', coalesce(member_house_number::text, '')), 'C') || - setweight(to_tsvector('simple', coalesce(member_postal_code::text, '')), 'C') || - setweight(to_tsvector('simple', coalesce(custom_values_text, '')), 'C') - WHERE id = member_id_val; - - RETURN COALESCE(NEW, OLD); - END - $$ LANGUAGE plpgsql; - """) - - # Create trigger on custom_field_values table - execute(""" - CREATE TRIGGER update_member_search_vector_on_custom_field_value_change - AFTER INSERT OR UPDATE OR DELETE ON custom_field_values - FOR EACH ROW - EXECUTE FUNCTION update_member_search_vector_from_custom_field_value() - """) - - # Update existing search_vector values for all members - execute(""" - UPDATE members m - SET search_vector = - setweight(to_tsvector('simple', coalesce(m.first_name, '')), 'A') || - setweight(to_tsvector('simple', coalesce(m.last_name, '')), 'A') || - setweight(to_tsvector('simple', coalesce(m.email, '')), 'B') || - setweight(to_tsvector('simple', coalesce(m.phone_number, '')), 'C') || - setweight(to_tsvector('simple', coalesce(m.join_date::text, '')), 'D') || - setweight(to_tsvector('simple', coalesce(m.exit_date::text, '')), 'D') || - setweight(to_tsvector('simple', coalesce(m.notes, '')), 'B') || - setweight(to_tsvector('simple', coalesce(m.city, '')), 'C') || - setweight(to_tsvector('simple', coalesce(m.street, '')), 'C') || - setweight(to_tsvector('simple', coalesce(m.house_number::text, '')), 'C') || - setweight(to_tsvector('simple', coalesce(m.postal_code::text, '')), 'C') || - setweight(to_tsvector('simple', coalesce( - (SELECT string_agg( - CASE - WHEN value ? '_union_value' THEN value->>'_union_value' - WHEN value ? 'value' THEN value->>'value' - ELSE '' - END, - ' ' - ) - FROM custom_field_values - WHERE member_id = m.id AND value IS NOT NULL), - '' - )), 'C') - """) - end - - def down do - # Drop trigger on custom_field_values - execute( - "DROP TRIGGER IF EXISTS update_member_search_vector_on_custom_field_value_change ON custom_field_values" - ) - - # Drop trigger function - execute("DROP FUNCTION IF EXISTS update_member_search_vector_from_custom_field_value()") - - # Restore original trigger function without custom_field_values - execute(""" - CREATE OR REPLACE FUNCTION members_search_vector_trigger() RETURNS trigger AS $$ - BEGIN - NEW.search_vector := - setweight(to_tsvector('simple', coalesce(NEW.first_name, '')), 'A') || - setweight(to_tsvector('simple', coalesce(NEW.last_name, '')), 'A') || - setweight(to_tsvector('simple', coalesce(NEW.email, '')), 'B') || - setweight(to_tsvector('simple', coalesce(NEW.phone_number, '')), 'C') || - setweight(to_tsvector('simple', coalesce(NEW.join_date::text, '')), 'D') || - setweight(to_tsvector('simple', coalesce(NEW.exit_date::text, '')), 'D') || - setweight(to_tsvector('simple', coalesce(NEW.notes, '')), 'B') || - setweight(to_tsvector('simple', coalesce(NEW.city, '')), 'C') || - setweight(to_tsvector('simple', coalesce(NEW.street, '')), 'C') || - setweight(to_tsvector('simple', coalesce(NEW.house_number::text, '')), 'C') || - setweight(to_tsvector('simple', coalesce(NEW.postal_code::text, '')), 'C'); - RETURN NEW; - END - $$ LANGUAGE plpgsql; - """) - - # Update existing search_vector values to remove custom_field_values - execute(""" - UPDATE members m - SET search_vector = - setweight(to_tsvector('simple', coalesce(m.first_name, '')), 'A') || - setweight(to_tsvector('simple', coalesce(m.last_name, '')), 'A') || - setweight(to_tsvector('simple', coalesce(m.email, '')), 'B') || - setweight(to_tsvector('simple', coalesce(m.phone_number, '')), 'C') || - setweight(to_tsvector('simple', coalesce(m.join_date::text, '')), 'D') || - setweight(to_tsvector('simple', coalesce(m.exit_date::text, '')), 'D') || - setweight(to_tsvector('simple', coalesce(m.notes, '')), 'B') || - setweight(to_tsvector('simple', coalesce(m.city, '')), 'C') || - setweight(to_tsvector('simple', coalesce(m.street, '')), 'C') || - setweight(to_tsvector('simple', coalesce(m.house_number::text, '')), 'C') || - setweight(to_tsvector('simple', coalesce(m.postal_code::text, '')), 'C') - """) - end -end diff --git a/priv/resource_snapshots/repo/members/20251204123714.json b/priv/resource_snapshots/repo/members/20251204123714.json deleted file mode 100644 index 8f3bf6c..0000000 --- a/priv/resource_snapshots/repo/members/20251204123714.json +++ /dev/null @@ -1,202 +0,0 @@ -{ - "attributes": [ - { - "allow_nil?": false, - "default": "fragment(\"uuid_generate_v7()\")", - "generated?": false, - "precision": null, - "primary_key?": true, - "references": null, - "scale": null, - "size": null, - "source": "id", - "type": "uuid" - }, - { - "allow_nil?": false, - "default": "nil", - "generated?": false, - "precision": null, - "primary_key?": false, - "references": null, - "scale": null, - "size": null, - "source": "first_name", - "type": "text" - }, - { - "allow_nil?": false, - "default": "nil", - "generated?": false, - "precision": null, - "primary_key?": false, - "references": null, - "scale": null, - "size": null, - "source": "last_name", - "type": "text" - }, - { - "allow_nil?": false, - "default": "nil", - "generated?": false, - "precision": null, - "primary_key?": false, - "references": null, - "scale": null, - "size": null, - "source": "email", - "type": "text" - }, - { - "allow_nil?": true, - "default": "nil", - "generated?": false, - "precision": null, - "primary_key?": false, - "references": null, - "scale": null, - "size": null, - "source": "paid", - "type": "boolean" - }, - { - "allow_nil?": true, - "default": "nil", - "generated?": false, - "precision": null, - "primary_key?": false, - "references": null, - "scale": null, - "size": null, - "source": "phone_number", - "type": "text" - }, - { - "allow_nil?": true, - "default": "nil", - "generated?": false, - "precision": null, - "primary_key?": false, - "references": null, - "scale": null, - "size": null, - "source": "join_date", - "type": "date" - }, - { - "allow_nil?": true, - "default": "nil", - "generated?": false, - "precision": null, - "primary_key?": false, - "references": null, - "scale": null, - "size": null, - "source": "exit_date", - "type": "date" - }, - { - "allow_nil?": true, - "default": "nil", - "generated?": false, - "precision": null, - "primary_key?": false, - "references": null, - "scale": null, - "size": null, - "source": "notes", - "type": "text" - }, - { - "allow_nil?": true, - "default": "nil", - "generated?": false, - "precision": null, - "primary_key?": false, - "references": null, - "scale": null, - "size": null, - "source": "city", - "type": "text" - }, - { - "allow_nil?": true, - "default": "nil", - "generated?": false, - "precision": null, - "primary_key?": false, - "references": null, - "scale": null, - "size": null, - "source": "street", - "type": "text" - }, - { - "allow_nil?": true, - "default": "nil", - "generated?": false, - "precision": null, - "primary_key?": false, - "references": null, - "scale": null, - "size": null, - "source": "house_number", - "type": "text" - }, - { - "allow_nil?": true, - "default": "nil", - "generated?": false, - "precision": null, - "primary_key?": false, - "references": null, - "scale": null, - "size": null, - "source": "postal_code", - "type": "text" - }, - { - "allow_nil?": true, - "default": "nil", - "generated?": false, - "precision": null, - "primary_key?": false, - "references": null, - "scale": null, - "size": null, - "source": "search_vector", - "type": "tsvector" - } - ], - "base_filter": null, - "check_constraints": [], - "custom_indexes": [], - "custom_statements": [], - "has_create_action": true, - "hash": "247CACFA5C8FD24BDD553252E9BBF489E8FE54F60704383B6BE66C616D203A65", - "identities": [ - { - "all_tenants?": false, - "base_filter": null, - "index_name": "members_unique_email_index", - "keys": [ - { - "type": "atom", - "value": "email" - } - ], - "name": "unique_email", - "nils_distinct?": true, - "where": null - } - ], - "multitenancy": { - "attribute": null, - "global": null, - "strategy": null - }, - "repo": "Elixir.Mv.Repo", - "schema": null, - "table": "members" -} \ No newline at end of file diff --git a/test/membership/member_search_with_custom_fields_test.exs b/test/membership/member_search_with_custom_fields_test.exs deleted file mode 100644 index 6711df8..0000000 --- a/test/membership/member_search_with_custom_fields_test.exs +++ /dev/null @@ -1,702 +0,0 @@ -defmodule Mv.Membership.MemberSearchWithCustomFieldsTest do - @moduledoc """ - Tests for full-text search including custom_field_values. - - Tests verify that custom field values are included in the search_vector - and can be found through the fuzzy_search functionality. - """ - use Mv.DataCase, async: false - - alias Mv.Membership.{CustomField, CustomFieldValue, Member} - - setup do - # Create test members - {:ok, member1} = - Member - |> Ash.Changeset.for_create(:create_member, %{ - first_name: "Alice", - last_name: "Anderson", - email: "alice@example.com" - }) - |> Ash.create() - - {:ok, member2} = - Member - |> Ash.Changeset.for_create(:create_member, %{ - first_name: "Bob", - last_name: "Brown", - email: "bob@example.com" - }) - |> Ash.create() - - {:ok, member3} = - Member - |> Ash.Changeset.for_create(:create_member, %{ - first_name: "Charlie", - last_name: "Clark", - email: "charlie@example.com" - }) - |> Ash.create() - - # Create custom fields for different types - {:ok, string_field} = - CustomField - |> Ash.Changeset.for_create(:create, %{ - name: "membership_number", - value_type: :string - }) - |> Ash.create() - - {:ok, integer_field} = - CustomField - |> Ash.Changeset.for_create(:create, %{ - name: "member_id_number", - value_type: :integer - }) - |> Ash.create() - - {:ok, email_field} = - CustomField - |> Ash.Changeset.for_create(:create, %{ - name: "secondary_email", - value_type: :email - }) - |> Ash.create() - - {:ok, date_field} = - CustomField - |> Ash.Changeset.for_create(:create, %{ - name: "birthday", - value_type: :date - }) - |> Ash.create() - - {:ok, boolean_field} = - CustomField - |> Ash.Changeset.for_create(:create, %{ - name: "newsletter", - value_type: :boolean - }) - |> Ash.create() - - %{ - member1: member1, - member2: member2, - member3: member3, - string_field: string_field, - integer_field: integer_field, - email_field: email_field, - date_field: date_field, - boolean_field: boolean_field - } - end - - describe "search with custom field values" do - test "finds member by string custom field value", %{ - member1: member1, - string_field: string_field - } do - # Create custom field value - {:ok, _cfv} = - CustomFieldValue - |> Ash.Changeset.for_create(:create, %{ - member_id: member1.id, - custom_field_id: string_field.id, - value: %{"_union_type" => "string", "_union_value" => "MEMBER12345"} - }) - |> Ash.create() - - # Force search_vector update by reloading member - {:ok, _updated_member} = - member1 - |> Ash.Changeset.for_update(:update_member, %{}) - |> Ash.update() - - # Search for the custom field value - results = - Member - |> Member.fuzzy_search(%{query: "MEMBER12345"}) - |> Ash.read!() - - assert length(results) == 1 - assert List.first(results).id == member1.id - end - - test "finds member by integer custom field value", %{ - member1: member1, - integer_field: integer_field - } do - # Create custom field value - {:ok, _cfv} = - CustomFieldValue - |> Ash.Changeset.for_create(:create, %{ - member_id: member1.id, - custom_field_id: integer_field.id, - value: %{"_union_type" => "integer", "_union_value" => 42_424} - }) - |> Ash.create() - - # Force search_vector update - {:ok, _updated_member} = - member1 - |> Ash.Changeset.for_update(:update_member, %{}) - |> Ash.update() - - # Search for the custom field value - results = - Member - |> Member.fuzzy_search(%{query: "42424"}) - |> Ash.read!() - - assert length(results) == 1 - assert List.first(results).id == member1.id - end - - test "finds member by email custom field value", %{ - member1: member1, - email_field: email_field - } do - # Create custom field value - {:ok, _cfv} = - CustomFieldValue - |> Ash.Changeset.for_create(:create, %{ - member_id: member1.id, - custom_field_id: email_field.id, - value: %{"_union_type" => "email", "_union_value" => "alice.secondary@example.com"} - }) - |> Ash.create() - - # Force search_vector update - {:ok, _updated_member} = - member1 - |> Ash.Changeset.for_update(:update_member, %{}) - |> Ash.update() - - # Search for partial custom field value (should work via FTS or custom field filter) - results = - Member - |> Member.fuzzy_search(%{query: "alice.secondary"}) - |> Ash.read!() - - assert length(results) == 1 - assert List.first(results).id == member1.id - - # Search for full email address (should work via custom field filter LIKE) - results_full = - Member - |> Member.fuzzy_search(%{query: "alice.secondary@example.com"}) - |> Ash.read!() - - assert length(results_full) == 1 - assert List.first(results_full).id == member1.id - - # Search for domain part (should work via FTS or custom field filter) - # Note: May return multiple results if other members have same domain - results_domain = - Member - |> Member.fuzzy_search(%{query: "example.com"}) - |> Ash.read!() - - # Verify that member1 is in the results (may have other members too) - ids = Enum.map(results_domain, & &1.id) - assert member1.id in ids - end - - test "finds member by date custom field value", %{ - member1: member1, - date_field: date_field - } do - # Create custom field value - {:ok, _cfv} = - CustomFieldValue - |> Ash.Changeset.for_create(:create, %{ - member_id: member1.id, - custom_field_id: date_field.id, - value: %{"_union_type" => "date", "_union_value" => ~D[1990-05-15]} - }) - |> Ash.create() - - # Force search_vector update - {:ok, _updated_member} = - member1 - |> Ash.Changeset.for_update(:update_member, %{}) - |> Ash.update() - - # Search for the custom field value (date is stored as text in search_vector) - results = - Member - |> Member.fuzzy_search(%{query: "1990-05-15"}) - |> Ash.read!() - - assert length(results) == 1 - assert List.first(results).id == member1.id - end - - test "finds member by boolean custom field value", %{ - member1: member1, - boolean_field: boolean_field - } do - # Create custom field value - {:ok, _cfv} = - CustomFieldValue - |> Ash.Changeset.for_create(:create, %{ - member_id: member1.id, - custom_field_id: boolean_field.id, - value: %{"_union_type" => "boolean", "_union_value" => true} - }) - |> Ash.create() - - # Force search_vector update - {:ok, _updated_member} = - member1 - |> Ash.Changeset.for_update(:update_member, %{}) - |> Ash.update() - - # Search for the custom field value (boolean is stored as "true" or "false" text) - results = - Member - |> Member.fuzzy_search(%{query: "true"}) - |> Ash.read!() - - # Note: "true" might match other things, so we check that member1 is in results - assert Enum.any?(results, fn m -> m.id == member1.id end) - end - - test "custom field value update triggers search_vector update", %{ - member1: member1, - string_field: string_field - } do - # Create initial custom field value - {:ok, cfv} = - CustomFieldValue - |> Ash.Changeset.for_create(:create, %{ - member_id: member1.id, - custom_field_id: string_field.id, - value: %{"_union_type" => "string", "_union_value" => "OLDVALUE"} - }) - |> Ash.create() - - # Force search_vector update - {:ok, _updated_member} = - member1 - |> Ash.Changeset.for_update(:update_member, %{}) - |> Ash.update() - - # Update custom field value - {:ok, _updated_cfv} = - cfv - |> Ash.Changeset.for_update(:update, %{ - value: %{"_union_type" => "string", "_union_value" => "NEWVALUE123"} - }) - |> Ash.update() - - # Search for the new value - results = - Member - |> Member.fuzzy_search(%{query: "NEWVALUE123"}) - |> Ash.read!() - - assert length(results) == 1 - assert List.first(results).id == member1.id - - # Old value should not be found - old_results = - Member - |> Member.fuzzy_search(%{query: "OLDVALUE"}) - |> Ash.read!() - - refute Enum.any?(old_results, fn m -> m.id == member1.id end) - end - - test "custom field value delete triggers search_vector update", %{ - member1: member1, - string_field: string_field - } do - # Create custom field value - {:ok, cfv} = - CustomFieldValue - |> Ash.Changeset.for_create(:create, %{ - member_id: member1.id, - custom_field_id: string_field.id, - value: %{"_union_type" => "string", "_union_value" => "TOBEDELETED"} - }) - |> Ash.create() - - # Force search_vector update - {:ok, _updated_member} = - member1 - |> Ash.Changeset.for_update(:update_member, %{}) - |> Ash.update() - - # Verify it's searchable - results = - Member - |> Member.fuzzy_search(%{query: "TOBEDELETED"}) - |> Ash.read!() - - assert length(results) == 1 - assert List.first(results).id == member1.id - - # Delete custom field value - assert :ok = Ash.destroy(cfv) - - # Value should no longer be found - deleted_results = - Member - |> Member.fuzzy_search(%{query: "TOBEDELETED"}) - |> Ash.read!() - - refute Enum.any?(deleted_results, fn m -> m.id == member1.id end) - end - - test "custom field value create triggers search_vector update", %{ - member1: member1, - string_field: string_field - } do - # Create custom field value (trigger should update search_vector automatically) - {:ok, _cfv} = - CustomFieldValue - |> Ash.Changeset.for_create(:create, %{ - member_id: member1.id, - custom_field_id: string_field.id, - value: %{"_union_type" => "string", "_union_value" => "AUTOUPDATE"} - }) - |> Ash.create() - - # Search should find it immediately (trigger should have updated search_vector) - results = - Member - |> Member.fuzzy_search(%{query: "AUTOUPDATE"}) - |> Ash.read!() - - assert length(results) == 1 - assert List.first(results).id == member1.id - end - - test "member update includes custom field values in search_vector", %{ - member1: member1, - string_field: string_field - } do - # Create custom field value - {:ok, _cfv} = - CustomFieldValue - |> Ash.Changeset.for_create(:create, %{ - member_id: member1.id, - custom_field_id: string_field.id, - value: %{"_union_type" => "string", "_union_value" => "MEMBERUPDATE"} - }) - |> Ash.create() - - # Update member (should trigger search_vector update including custom fields) - {:ok, _updated_member} = - member1 - |> Ash.Changeset.for_update(:update_member, %{notes: "Updated notes"}) - |> Ash.update() - - # Search should find the custom field value - results = - Member - |> Member.fuzzy_search(%{query: "MEMBERUPDATE"}) - |> Ash.read!() - - assert length(results) == 1 - assert List.first(results).id == member1.id - end - - test "multiple custom field values are all searchable", %{ - member1: member1, - string_field: string_field, - integer_field: integer_field, - email_field: email_field - } do - # Create multiple custom field values - {:ok, _cfv1} = - CustomFieldValue - |> Ash.Changeset.for_create(:create, %{ - member_id: member1.id, - custom_field_id: string_field.id, - value: %{"_union_type" => "string", "_union_value" => "MULTI1"} - }) - |> Ash.create() - - {:ok, _cfv2} = - CustomFieldValue - |> Ash.Changeset.for_create(:create, %{ - member_id: member1.id, - custom_field_id: integer_field.id, - value: %{"_union_type" => "integer", "_union_value" => 99_999} - }) - |> Ash.create() - - {:ok, _cfv3} = - CustomFieldValue - |> Ash.Changeset.for_create(:create, %{ - member_id: member1.id, - custom_field_id: email_field.id, - value: %{"_union_type" => "email", "_union_value" => "multi@test.com"} - }) - |> Ash.create() - - # Force search_vector update - {:ok, _updated_member} = - member1 - |> Ash.Changeset.for_update(:update_member, %{}) - |> Ash.update() - - # All values should be searchable - results1 = - Member - |> Member.fuzzy_search(%{query: "MULTI1"}) - |> Ash.read!() - - assert Enum.any?(results1, fn m -> m.id == member1.id end) - - results2 = - Member - |> Member.fuzzy_search(%{query: "99999"}) - |> Ash.read!() - - assert Enum.any?(results2, fn m -> m.id == member1.id end) - - results3 = - Member - |> Member.fuzzy_search(%{query: "multi@test.com"}) - |> Ash.read!() - - assert Enum.any?(results3, fn m -> m.id == member1.id end) - end - - test "finds member by custom field value with numbers in text field (e.g. phone number)", %{ - member1: member1, - string_field: string_field - } do - # Create custom field value with numbers and text (like phone number or ID) - {:ok, _cfv} = - CustomFieldValue - |> Ash.Changeset.for_create(:create, %{ - member_id: member1.id, - custom_field_id: string_field.id, - value: %{"_union_type" => "string", "_union_value" => "M-123-456"} - }) - |> Ash.create() - - # Force search_vector update - {:ok, _updated_member} = - member1 - |> Ash.Changeset.for_update(:update_member, %{}) - |> Ash.update() - - # Search for full value (should work via search_vector) - results_full = - Member - |> Member.fuzzy_search(%{query: "M-123-456"}) - |> Ash.read!() - - assert Enum.any?(results_full, fn m -> m.id == member1.id end), - "Full value search should find member via search_vector" - - # Note: Partial substring search may require additional implementation - # For now, we test that the full value is searchable, which is the primary use case - # Substring matching for custom fields may need to be implemented separately - end - - test "finds member by phone number in Emergency Contact custom field", %{ - member1: member1 - } do - # Create Emergency Contact custom field - {:ok, emergency_contact_field} = - CustomField - |> Ash.Changeset.for_create(:create, %{ - name: "Emergency Contact", - value_type: :string - }) - |> Ash.create() - - # Create custom field value with phone number - phone_number = "+49 123 456789" - - {:ok, _cfv} = - CustomFieldValue - |> Ash.Changeset.for_create(:create, %{ - member_id: member1.id, - custom_field_id: emergency_contact_field.id, - value: %{"_union_type" => "string", "_union_value" => phone_number} - }) - |> Ash.create() - - # Force search_vector update - {:ok, _updated_member} = - member1 - |> Ash.Changeset.for_update(:update_member, %{}) - |> Ash.update() - - # Search for full phone number (should work via search_vector) - results_full = - Member - |> Member.fuzzy_search(%{query: phone_number}) - |> Ash.read!() - - assert Enum.any?(results_full, fn m -> m.id == member1.id end), - "Full phone number search should find member via search_vector" - - # Note: Partial substring search may require additional implementation - # For now, we test that the full phone number is searchable, which is the primary use case - # Substring matching for custom fields may need to be implemented separately - end - end - - describe "custom field substring search (ILIKE)" do - test "finds member by prefix of custom field value", %{ - member1: member1, - string_field: string_field - } do - # Create custom field value with a distinct word - {:ok, _cfv} = - CustomFieldValue - |> Ash.Changeset.for_create(:create, %{ - member_id: member1.id, - custom_field_id: string_field.id, - value: %{"_union_type" => "string", "_union_value" => "Premium"} - }) - |> Ash.create() - - # Test prefix searches - should all find the member - for prefix <- ["Premium", "Premiu", "Premi", "Prem", "Pre"] do - results = - Member - |> Member.fuzzy_search(%{query: prefix}) - |> Ash.read!() - - assert Enum.any?(results, fn m -> m.id == member1.id end), - "Prefix '#{prefix}' should find member with custom field 'Premium'" - end - end - - test "custom field search is case-insensitive", %{ - member1: member1, - string_field: string_field - } do - # Create custom field value - {:ok, _cfv} = - CustomFieldValue - |> Ash.Changeset.for_create(:create, %{ - member_id: member1.id, - custom_field_id: string_field.id, - value: %{"_union_type" => "string", "_union_value" => "GoldMember"} - }) - |> Ash.create() - - # Test case variations - should all find the member - for variant <- [ - "GoldMember", - "goldmember", - "GOLDMEMBER", - "GoLdMeMbEr", - "gold", - "GOLD", - "Gold" - ] do - results = - Member - |> Member.fuzzy_search(%{query: variant}) - |> Ash.read!() - - assert Enum.any?(results, fn m -> m.id == member1.id end), - "Case variant '#{variant}' should find member with custom field 'GoldMember'" - end - end - - test "finds member by suffix/middle of custom field value", %{ - member1: member1, - string_field: string_field - } do - # Create custom field value - {:ok, _cfv} = - CustomFieldValue - |> Ash.Changeset.for_create(:create, %{ - member_id: member1.id, - custom_field_id: string_field.id, - value: %{"_union_type" => "string", "_union_value" => "ActiveMember"} - }) - |> Ash.create() - - # Test suffix and middle substring searches - for substring <- ["Member", "ember", "tiveMem", "ctive"] do - results = - Member - |> Member.fuzzy_search(%{query: substring}) - |> Ash.read!() - - assert Enum.any?(results, fn m -> m.id == member1.id end), - "Substring '#{substring}' should find member with custom field 'ActiveMember'" - end - end - - test "finds correct member among multiple with different custom field values", %{ - member1: member1, - member2: member2, - member3: member3, - string_field: string_field - } do - # Create different custom field values for each member - {:ok, _cfv1} = - CustomFieldValue - |> Ash.Changeset.for_create(:create, %{ - member_id: member1.id, - custom_field_id: string_field.id, - value: %{"_union_type" => "string", "_union_value" => "Beginner"} - }) - |> Ash.create() - - {:ok, _cfv2} = - CustomFieldValue - |> Ash.Changeset.for_create(:create, %{ - member_id: member2.id, - custom_field_id: string_field.id, - value: %{"_union_type" => "string", "_union_value" => "Advanced"} - }) - |> Ash.create() - - {:ok, _cfv3} = - CustomFieldValue - |> Ash.Changeset.for_create(:create, %{ - member_id: member3.id, - custom_field_id: string_field.id, - value: %{"_union_type" => "string", "_union_value" => "Expert"} - }) - |> Ash.create() - - # Search for "Begin" - should only find member1 - results_begin = - Member - |> Member.fuzzy_search(%{query: "Begin"}) - |> Ash.read!() - - assert length(results_begin) == 1 - assert List.first(results_begin).id == member1.id - - # Search for "Advan" - should only find member2 - results_advan = - Member - |> Member.fuzzy_search(%{query: "Advan"}) - |> Ash.read!() - - assert length(results_advan) == 1 - assert List.first(results_advan).id == member2.id - - # Search for "Exper" - should only find member3 - results_exper = - Member - |> Member.fuzzy_search(%{query: "Exper"}) - |> Ash.read!() - - assert length(results_exper) == 1 - assert List.first(results_exper).id == member3.id - end - - # Note: Legacy format (type/value) is supported via the SQL ILIKE query on value->>'value' - # This is tested implicitly by the migration trigger which handles both formats. - # The Ash union type only accepts the new format (_union_type/_union_value) for creation, - # but the search works on existing legacy data. - end -end