[FEATURE] Support ScaNN/FastScan and re-ranking for faiss Engine

[rishabhagrawal1]

trafficstars

As per mentioned in the paper "Product quantization for nearest neighbor search" and implementation using registers to store look-up tables are providing much better accuracy and latency results for recommendation task.

ScaNN performance

FastScan performance Comparison with HSNW: without reranking, 4-bit PQ is able to do up to 1M QPS. With re-ranking, it is at 280k QPS with 1-recall@1 = 0.9, which it 2x faster than HNSW's 140k QPS. It also uses 2.7x less memory because the graph structure does not need to be stored.

What solution would you like? Supporting either of the two algorithms along with re-ranking support.

What alternatives have you considered? We tried configurations suggested in billion vector blog for HNSW.

[heemin32]

Could you share how you got those result in the graph? Like test environment setup to reproduce?

[rishabhagrawal1]

ScaNN results were published by ann-benchmarks and referred by ScaNN here.