inventree
[FR] Enable accent-insensitive search
Please verify that this feature request has NOT been suggested before.
- [x] I checked and didn't find a similar feature request
Problem statement
Hello, my team and I are using Inventree with french speaking users and sometimes someone forget to properly accentuate a part name (for example screen / "écran" / "ecran") leading to confusion when someone else search this part. It would be great if the search bar and search functions could be modified (or configured) to be accent-insensitive.
Suggested solution
I am a rather experienced Python developer but I've never work with the Django framework before. I'm ready to implement this feature by myself but some pointers would be really useful. After some time looking up the code it seems that I need to extend the django framework view classes to request an unaccentuated search in Postgres (which may need this feature https://www.postgresql.org/docs/current/unaccent.html).
Any help would be appreciated ! Thanks.
Describe alternatives you've considered
An alternative I considered was to remove accentuated letters in the search api endpoint but it would work only one way : a search about "écran" would match an "ecran" part, but not the other way around.
Another way of solving my problem is to enable a more "fuzzy" search by using trigram matching, postgres also seems to support this: https://www.postgresql.org/docs/current/pgtrgm.html
Examples of other systems
No response
Do you want to develop this?
- [x] I want to develop this.
We currently support sqlite, mysql/mariadb and postgresql - whatever solution you come up with should work for all 3, not just one backend.
I do not think that there is a good general-purpose way to do this.
- Sqlite doesn't support this
- Postgresql and mysql both seem to, but requires low-level database configuration changes and extra collation / indexing
If you can propose a technical solution, perhaps?
I am aware of that, however implementing a DB specific solution would probably lead to confusion when it only works for one thing; also replacing the specific DB backend is something pretty common (eg for tracing, warehousing) so that should be considered
My comment was directed at @JDRobotter - perhaps they can suggest a general purpose solution. Otherwise I am not supportive
Okay, I do not realized Inventree was supporting all those DBs. In this case a more general solution is in order, I will think about that. Thanks for your comments.
I did some digging around a way to transform a search input into a regexp which may include any potential accentuated character.
For example transforming écran into [eéèê]cr[aà]n.
But this option would be pretty heavy on the server load and more importantly, near impossible if we take into account all possible languages. Unicode as around 100 diacritics and every character potentially need to be checked.
I think the best solution to add a field "name_normalized" to every "name" field which may be searched. This field would contain the normalized unicode version of the "name" string which can be done with the python unicode.normalize("NFD", string) function.
The same transformation has to be applied when searching for a string and listing entries against this "name_normalized" field instead of "name".
I guess this will induce a lot of refactoring, but with some help I can work on it if you think this is a viable solution !
That sounds like a lot of work to include and maintain, not sure if it is worth it.
What about an optionally added meilisearch? We could fill the meilisearch with part create / update trigger and then use this to query the db when searching.
We already support regex search so at worst you can construct a regex yourself. A big advantage of this approach is that it supports all searched fields - a manual indexing or duplicate field would only apply to selected fields.
Perhaps (in a future release) we can offload some search functionality to plugins - then we don't have to think about optional bundling of any other packages. We just need to determine the appropriate stubs to expose to the plugin interface.
Closing this one out, unfortunately I do not think this will be achievable in a reliable, generic way