sciencefair
sciencefair copied to clipboard
Distributed search index
Just an issue to track ideas on how to distribute the search index generated from the initialization step
In the simplest case we could just use hyperdrive as the kvstore instead of leveldb - but the latency involved in the lookups for the inverted index would probably be a huge bottleneck.
Worth trying as a POC I think, then we can optimise from there.
Some references:
- https://en.wikipedia.org/wiki/YaCy
- https://github.com/beniz/seeks (abandoned)
- http://www.faroo.com/hp/p2p/p2p.html
I am in the process of implementing this on top of hyperdb. A lot of the hard problems are already solved by @mafintosh in that module. There are some tricky things to do with implementing an efficient full text search index on top it it - I hope to have them all ironed out with an initial release in the next few days.