Implement fuzzy search #187

Merged
carla merged 10 commits from feature/162_fuzzy_search into main 2025-11-12 13:10:32 +01:00
Owner

Description of the implemented changes

  • added trigram migration to enable fuzzy search as described in the issue
  • added combination of fuzzy search, contains and Ilike for specific fields to members ressource
  • Use fuzzy search in member live view

The changes were:

  • Bugfixing
  • New Feature
  • Breaking Change
  • Refactoring

What has been changed?

Definition of Done

Code Quality

  • No new technical depths
  • Linting passed
  • Documentation is added were needed

Accessibility

  • New elements are properly defined with html-tags
  • Colour contrast follows WCAG criteria
  • Aria labels are added when needed
  • Everything is accessible by keyboard
  • Tab-Order is comprehensible
  • All interactive elements have a visible focus

Testing

  • Tests for new code are written
  • All tests pass
  • axe-core dev tools show no critical or major issues

Additional Notes

## Description of the implemented changes - added trigram migration to enable fuzzy search as described in the issue - added combination of fuzzy search, contains and Ilike for specific fields to members ressource - Use fuzzy search in member live view The changes were: - [ ] Bugfixing - [x] New Feature - [ ] Breaking Change - [ ] Refactoring <!--- Describe the goal of the PR in a few words --> ## What has been changed? <!--- List the things you changed --> ## Definition of Done ### Code Quality - [x] No new technical depths - [x] Linting passed - [x] Documentation is added were needed ### Accessibility - [x] New elements are properly defined with html-tags - [x] Colour contrast follows WCAG criteria - [x] Aria labels are added when needed - [x] Everything is accessible by keyboard - [x] Tab-Order is comprehensible - [x] All interactive elements have a visible focus ### Testing - [x] Tests for new code are written - [x] All tests pass - [x] axe-core dev tools show no critical or major issues ## Additional Notes <!--- Add any additional information for the reviewers here -->
carla added 6 commits 2025-10-30 17:19:23 +01:00
carla added 1 commit 2025-10-30 17:20:14 +01:00
formatting
All checks were successful
continuous-integration/drone/push Build is passing
0c75776915
requested reviews from moritz, simon 2025-10-30 17:21:10 +01:00
carla added this to the I can search through the list of members -fulltext milestone 2025-10-30 17:21:19 +01:00
carla added this to the Sprint 8 - 23.10 - 20.11 project 2025-10-30 17:21:24 +01:00
Author
Owner

Not sure about accessibility with search on typing, so I created that issue: #188

Not sure about accessibility with search on typing, so I created that issue: https://git.local-it.org/local-it/mitgliederverwaltung/issues/188
moritz requested changes 2025-11-05 17:50:54 +01:00
moritz left a comment
Owner

Overall this looks very good 🚀
Filtering for emails does not work and I added some small suggestions.

Further some tests like these could be useful:
blank character handling
special character handling (@, ., -)
unicode character handling (umlauts ö,ä,ü)
very long search strings

Overall this looks very good 🚀 Filtering for emails does not work and I added some small suggestions. Further some tests like these could be useful: blank character handling special character handling (@, ., -) unicode character handling (umlauts ö,ä,ü) very long search strings
@ -109,2 +123,4 @@
end
end
read :search do
Owner

some moduledocs would be nice for the public search function.

some moduledocs would be nice for the public `search` function.
carla marked this conversation as resolved
@ -111,0 +130,4 @@
prepare fn query, _ctx ->
q = Ash.Query.get_argument(query, :query) || ""
fields = Ash.Query.get_argument(query, :fields) || @default_fields
Owner

the fields argument is never used in the search function.

the `fields` argument is never used in the `search` function.
carla marked this conversation as resolved
@ -111,0 +131,4 @@
prepare fn query, _ctx ->
q = Ash.Query.get_argument(query, :query) || ""
fields = Ash.Query.get_argument(query, :fields) || @default_fields
threshold = Ash.Query.get_argument(query, :similarity_threshold) || 0.2
Owner

Maybe some documentation, what this 0.2 threshold means or where does it came from?

Maybe some documentation, what this `0.2` threshold means or where does it came from?
carla marked this conversation as resolved
@ -111,0 +137,4 @@
q2 = String.trim(q)
pat = "%" <> q2 <> "%"
# FTS as main filter and fuzzy search just fo first name, last name and strees
Owner

little typo: fo

little typo: `fo`
carla marked this conversation as resolved
@ -111,0 +151,4 @@
fragment("? % first_name", ^q2) or
fragment("? % last_name", ^q2) or
fragment("? % street", ^q2) or
fragment("word_similarity(?, first_name) > ?", ^q2, ^threshold) or
Owner

maybe the duplicate code for first name last name and street could be generated dynamically by the fields argument?

maybe the duplicate code for `first name` `last name` and `street` could be generated dynamically by the fields argument?
Author
Owner

I decided to omit fields now...but maybe during refactoring we could think about a dynamic way

I decided to omit fields now...but maybe during refactoring we could think about a dynamic way
carla marked this conversation as resolved
@ -111,0 +153,4 @@
fragment("? % street", ^q2) or
fragment("word_similarity(?, first_name) > ?", ^q2, ^threshold) or
fragment("word_similarity(?, last_name) > ?", ^q2, ^threshold) or
fragment("word_similarity(?, street) > ?", ^q2, ^threshold) or
Owner

filter for email is missing
adding this worked for me:

                fragment("? % email", ^q2) or
                fragment("word_similarity(?, email) > ?", ^q2, ^threshold) or
                fragment("similarity(email, ?) > ?", ^q2, ^threshold) or
filter for email is missing adding this worked for me: ``` fragment("? % email", ^q2) or fragment("word_similarity(?, email) > ?", ^q2, ^threshold) or fragment("similarity(email, ?) > ?", ^q2, ^threshold) or ```
Author
Owner

You're totally right, i forgot the email field. But I would favor for a simple contains ilike on the email field, because I think fuzzy search on the name already is enough? or what do you think?

You're totally right, i forgot the email field. But I would favor for a simple contains ilike on the email field, because I think fuzzy search on the name already is enough? or what do you think?
Author
Owner

I added it as ilike and not as fuzzy search. But we can create another issue if we see we need it :)

I added it as ilike and not as fuzzy search. But we can create another issue if we see we need it :)
carla marked this conversation as resolved
@ -197,1 +195,3 @@
|> filter(expr(fragment("search_vector @@ plainto_tsquery('simple', ?)", ^search_query)))
|> Mv.Membership.Member.fuzzy_search(%{
query: search_query,
fields: [:first_name, :last_name, :street]
Owner

these fields are passed to fuzzy_serach->search function but don't have any effect, as the search function ignores them.

these fields are passed to `fuzzy_serach`->`search` function but don't have any effect, as the `search` function ignores them.
carla marked this conversation as resolved
@ -0,0 +31,4 @@
Mv.Membership.Member
|> Mv.Membership.Member.fuzzy_search(%{
query: "john",
fields: [:first_name, :last_name, :email]
Owner

In this test the email field is given, but it only tests the first_name "John" and the last_name "Johnson". It would be good to explicitly test the email field.

In this test the `email` field is given, but it only tests the `first_name` "John" and the `last_name` "Johnson". It would be good to explicitly test the `email` field.
carla marked this conversation as resolved
carla added 2 commits 2025-11-12 11:56:00 +01:00
test: aded more tests for fuzzy search
All checks were successful
continuous-integration/drone/push Build is passing
44f88f1ddd
carla added 1 commit 2025-11-12 12:01:03 +01:00
Merge branch 'main' into feature/162_fuzzy_search
All checks were successful
continuous-integration/drone/push Build is passing
56516d78b6
carla merged commit 7305c63130 into main 2025-11-12 13:10:32 +01:00
carla deleted branch feature/162_fuzzy_search 2025-11-12 13:10:32 +01:00
carla self-assigned this 2025-11-12 13:10:47 +01:00
Sign in to join this conversation.
No description provided.