Alternative search backends

[mayel]

Hi, wondering if there's any plans/interest in supporting alternative search index software, such as MeiliSearch for example?

[simon987]

Hi, There are no plans in supporting other search backends right now. However if someone would like to create a PR for this I wouldn't mind integrating & maintaining that feature.

[vercas]

Would it be possible to also use a relational database for this, e.g. MySQL?
Elasticsearch uses rather high amounts of memory, it would be nice to have an option that makes more use of storage, than memory.
I suppose many people will want to use this project on consumer NAS devices, which can be stuffed with beefy disks but typically come with weak CPUs and little memory - none of which are upgradeable.

[simon987]

That wouldn't really be possible without very significant changes, and the search performance would probably suffer. You should be able to run Elasticsearch with <512MB of memory, and it doesn't need to run on the same host as the files themselves (or the web UI): You can run the Web UI on one host, serve the files another host and run Elasticsearch on a third host.

[anarchomoh]

Hi, thank you for this amazing software. Please consider https://manticoresearch.com/ as the search backend as it is fully open source.

[nick008a]

try https://github.com/zinclabs/zinc, maybe

[drewbitt]

ZincSearch could potentially work out of the box but currently it fails at version check with the /es/ compatibility endpoint at https://github.com/simon987/sist2/blob/4e1109c5286e54271f11e3ee72e00d111dba76ff/src/index/elastic.c#L430 returning null

[nick008a]

@drewbitt Does it work after removing this check?

[simon987]

No promises but I will attempt to make fully-embedded search index with SQLite FTS5 work. If it does work, the web server could run with only ~100mb or ram

[ghost]

That would be awesome!!

With SQlite, keeping the search engine and the sist2 index in sync will be a lot easier. (Maybe there is no need for a sist2 index anymore)

Does FTS5 support BM25 algorithm for better ranking?

[simon987]

With SQlite, keeping the search engine and the sist2 index in sync will be a lot easier. (Maybe there is no need for a sist2 index anymore)

You will still need to have at least two separate indices (the regular sist2 index and the search index). They probably can exist in the same database file. To search through multiple indices, it necessarily needs one central search index.

Does FTS5 support BM25 algorithm for better ranking?

yes

[youk]

FYI ES-compatible API · Issue #2653 · quickwit