LittleBigRefresh
Investigate improving text search
This issue is a bit of a ramble, just trying to lay down my thoughts somewhere as I research this.
Why
Currently, our solution for text search is less than ideal and often hides results if you search something missing in a word - appl instead of apple for instance.
Realm doesn't provide a good search implementation, and writing our own would be a bit of a disaster. So that leaves us with an external solution.
How
Obviously, adding an external service means that we still have to keep Realm text search in to avoid additional setup. Anyways, past that, we have two options.
Elasticsearch
The more popular choice according to Wikipedia, but it seems to be more enterprise than self-managed, since their whole website keeps trying to convince you to contact their sales team. I'm not sure if that's a good omen. Also licensed under a non-free license as of 2021 since Amazon started deploying Elasticsearch in a AWS service.
OpenSearch
A fork of Elasticsearch created as a reaction from their license change. Development is lead by AWS but the documentation page seems to be actually useful. Also heard whispers and rumors that it's a little bit more performant than Elasticsearch.
Don't know which we'll be going with since it seems like a bit of a pick your poison situation, but for now I'm leaning towards OpenSearch - right now I'm just trying to figure out what it does, since the scope of both seem to be more than just a search engine (also seems to be a bucket for logging/monitoring?)
Also, both seem cluster-friendly. I'll look into finding an operator for Kubernetes when we pick. We shouldn't have to pull Refresh into the cluster to use it, the service should be able to be accessed outside the cluster (possibly via node port).
Important thing to note, elasticsearch/opensearch is very memory heavy, on my server it eats >2gb of ram alone. I wonder if there are any other simple options available which could fit our "just a search API" needs better?
After some research, i see https://github.com/zincsearch/zincsearch which may be of interest, seems special tailored to be lightweight and only does application search, no log searching stuff.
I'll look into that. The bit regarding log searching was more of a note than an actual issue. I actually think it could be pretty useful.