big-ann-benchmarks

big-ann-benchmarks copied to clipboard

Team SISU entry for "BuddyPQ".

Hi Harsha, I believe everything is resolved. We are writing the report for delivery but I want to make a note about the docker tag used: "BuddyPQ" is a preprocessing/enrichment step for Product Quantization, and is really a clone of the faiss-t1 algorithm with some additions to incorporate this enrichment. The Docker.faissconda is used, but there is an identical Docker.pqbuddy tag which I created and did not test. I am hoping we can continue to use Docker.faissconda to illustrate the effectiveness of our solution, but let me know if this is a problem. Thank you so much!

Hi Harsha, I believe everything is resolved. We are writing the report for delivery but I want to make a note about the docker tag used: "BuddyPQ" is a preprocessing/enrichment step for Product Quantization, and is really a clone of the faiss-t1 algorithm with some additions to incorporate this enrichment. The Docker.faissconda is used, but there is an identical Docker.pqbuddy tag which I created and did not test. I am hoping we can continue to use Docker.faissconda to illustrate the effectiveness of our solution, but let me know if this is a problem. Thank you so much!

Thanks! I will test your code. Reusing algorithms is fine, look forward to the document describing the improvement. Please submit through CMT.

@binarymax What 1B scale benchmarks are you competing on? I only see entry for bigann-1B and here are the result I see on the public query set.

buddy-t1,"FaissIVFPQ(nprobe=4,quantizer_efSearch=8)",bigann-1B,10,157549.11314617124,4977.0801,1000000.0,44019788.0,279.40359117832185,0,0,0.32964000000000004 buddy-t1,"FaissIVFPQ(nprobe=8,quantizer_efSearch=8)",bigann-1B,10,102322.55861042668,9936.012,1000000.0,44019788.0,430.20609138202667,0,0,0.40381999999999996 buddy-t1,"FaissIVFPQ(nprobe=16,quantizer_efSearch=32)",bigann-1B,10,39976.97249280391,19686.379,1000000.0,44019788.0,1101.1286061725616,0,0,0.52309 buddy-t1,"FaissIVFPQ(nprobe=128,quantizer_efSearch=64)",bigann-1B,10,6701.528921136838,152193.2372,1000000.0,44019788.0,6568.618671652699,0,0,0.65124 buddy-t1,"FaissIVFPQ(nprobe=64,quantizer_efSearch=128)",bigann-1B,10,10801.512514193018,77058.5433,1000000.0,44019788.0,4075.335555289936,0,0,0.6298600000000001 buddy-t1,"FaissIVFPQ(nprobe=16,quantizer_efSearch=16)",bigann-1B,10,45570.299097243274,19746.5304,1000000.0,44019788.0,965.9754022255897,0,0,0.50839 buddy-t1,"FaissIVFPQ(nprobe=64,quantizer_efSearch=32)",bigann-1B,10,13176.702990555495,77328.6604,1000000.0,44019788.0,3340.728559454632,0,0,0.6176999999999999 buddy-t1,"FaissIVFPQ(nprobe=64,quantizer_efSearch=64)",bigann-1B,10,12224.406821155959,77170.1085,1000000.0,44019788.0,3600.975380156517,0,0,0.62765 buddy-t1,"FaissIVFPQ(nprobe=128,quantizer_efSearch=32)",bigann-1B,10,6824.489583401129,152079.9624,1000000.0,44019788.0,6450.268179332733,0,0,0.63652 buddy-t1,"FaissIVFPQ(nprobe=8,quantizer_efSearch=16)",bigann-1B,10,87359.49345996835,9893.5842,1000000.0,44019788.0,503.89243637466427,0,0,0.43705999999999995

@binarymax hey Max, just wanted to make sure you saw the question above from Harsha.

Hi @harsha-simhadri I only submitted results for BIGANN-1B, even though they do not outperform Faiss-t1 for the same dataset. Rather I was able to make improvements on 10M sized-data. Our report in CMT in the section "Statistical analysis for vector recomposition in FAISS (AKA “BuddyPQ”)" specifically outlines the approach and results. Thanks!