Distributed search index

[mafintosh]

Just an issue to track ideas on how to distribute the search index generated from the initialization step

[blahah]

In the simplest case we could just use hyperdrive as the kvstore instead of leveldb - but the latency involved in the lookups for the inverted index would probably be a huge bottleneck.

Worth trying as a POC I think, then we can optimise from there.

[aschrijver]

Some references:

• https://en.wikipedia.org/wiki/YaCy
• https://github.com/beniz/seeks (abandoned)
• http://www.faroo.com/hp/p2p/p2p.html

[blahah]

I am in the process of implementing this on top of hyperdb. A lot of the hard problems are already solved by @mafintosh in that module. There are some tricky things to do with implementing an efficient full text search index on top it it - I hope to have them all ironed out with an initial release in the next few days.