Harsha Vardhan Simhadri

> @harsha-simhadri that seems ok, what about msturing-1B ? algorithm dataset kst_ann_t1 bigann-1B 0.712190 deep-1B 0.751890 msspacev-1B 0.764542 msturing-1B 0.756419 kst_ann_t1,"FaissIVFPQ(nprobe=8,quantizer_efSearch=32)",msturing-1B,10,66824.77850163386,10069.2102,1000000.0,59792456.0,894.7647465608596,0,0,0.593676 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=256)",msturing-1B,10,9099.132505939227,54325.28295,1000000.0,59792456.0,6571.225988957957,0,0,0.766613 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=64)",msturing-1B,10,15615.604301120746,37104.68467,1000000.0,59792456.0,3829.0196682115366,0,0,0.713444 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=64)",msturing-1B,10,10939.395109902307,54358.29073,1000000.0,59792456.0,5465.79179189497,0,0,0.7304 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=128)",msturing-1B,10,10290.569001704127,54360.34076,1000000.0,59792456.0,5810.4130092416,0,0,0.756419 kst_ann_t1,"FaissIVFPQ(nprobe=64,quantizer_efSearch=64)",msturing-1B,10,8413.39183312635,71367.53788,1000000.0,59792456.0,7106.81936440628,0,0,0.740756 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=32)",msturing-1B,10,16490.77389595115,37066.83411,1000000.0,59792456.0,3625.812613602104,0,0,0.668101 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=96)",msturing-1B,10,10625.881372482654,54365.84108,1000000.0,59792456.0,5627.058490869445,0,0,0.747961 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=128)",msturing-1B,10,14235.256436820593,37117.2022,1000000.0,59792456.0,4200.30761408289,0,0,0.736908 kst_ann_t1,"FaissIVFPQ(nprobe=16,quantizer_efSearch=64)",msturing-1B,10,35700.37892090531,19349.76173,1000000.0,59792456.0,1674.8409346710584,0,0,0.675843

@DmitryKey is there a way to make this work without failing checks

Hmm, it did pass. Let me see what happened here.

@PwzXxm Here are the results for msspacev public query set for BBANN on Azure L8sv2 VM. Are they as expected? bbann,BBANN,msspacev-1B,10,1610.4208408102975,0.0,1000000.0,11878352.0,7375.930377318827,0.0,0.0,0.7577705007504434 bbann,BBANN,msspacev-1B,10,1820.4236647258076,0.0,1000000.0,11878352.0,6525.048113890081,0.0,0.0,0.747574703233729 bbann,BBANN,msspacev-1B,10,1755.869157475597,0.0,1000000.0,11878352.0,6764.941424836825,0.0,0.0,0.7542058943921408 bbann,BBANN,msspacev-1B,10,1936.7360372815363,0.0,1000000.0,11878352.0,6133.1806561893845,0.0,0.0,0.7458589166325555 bbann,BBANN,msspacev-1B,10,3912.993335161897,0.0,1000000.0,11878352.0,3035.617743905138,0.0,0.0,0.6773809523809524 bbann,BBANN,msspacev-1B,10,1821.776089651613,0.0,1000000.0,11878352.0,6520.20413895736,0.0,0.0,0.7500818665575113 bbann,BBANN,msspacev-1B,10,1518.3505425395547,0.0,1000000.0,11878352.0,7823.1947545739795,0.0,0.0,0.7602026197298404 bbann,BBANN,msspacev-1B,10,1702.433482388304,0.0,1000000.0,11878352.0,6977.2781861269195,0.0,0.0,0.7522240414790559 bbann,BBANN,msspacev-1B,10,1629.0401932767472,0.0,1000000.0,11878352.0,7291.626105373854,0.0,0.0,0.7563787692727522 bbann,BBANN,msspacev-1B,10,2070.9883234427366,0.0,1000000.0,11878352.0,5735.595833903039,0.0,0.0,0.7408104789193615

> > @PwzXxm Here are the results for msspacev public query set for BBANN on Azure L8sv2 VM. Are they as expected? > > bbann,BBANN,msspacev-1B,10,1610.4208408102975,0.0,1000000.0,11878352.0,7375.930377318827,0.0,0.0,0.7577705007504434 bbann,BBANN,msspacev-1B,10,1820.4236647258076,0.0,1000000.0,11878352.0,6525.048113890081,0.0,0.0,0.747574703233729 bbann,BBANN,msspacev-1B,10,1755.869157475597,0.0,1000000.0,11878352.0,6764.941424836825,0.0,0.0,0.7542058943921408 bbann,BBANN,msspacev-1B,10,1936.7360372815363,0.0,1000000.0,11878352.0,6133.1806561893845,0.0,0.0,0.7458589166325555 bbann,BBANN,msspacev-1B,10,3912.993335161897,0.0,1000000.0,11878352.0,3035.617743905138,0.0,0.0,0.6773809523809524 bbann,BBANN,msspacev-1B,10,1821.776089651613,0.0,1000000.0,11878352.0,6520.20413895736,0.0,0.0,0.7500818665575113...

> Just wondering did you count ties when getting the above results for msspacev and ssnpp? Not so on the public query datasets.

> > > > @PwzXxm Here are the results for msspacev public query set for BBANN on Azure L8sv2 VM. Are they as expected? > > > > > >...

@PwzXxm on text2image, I see the following. Does it align with what you see. bbann,BBANN,text2image-1B,10,1902.735310390915,0.0,1000000.0,29064836.0,15275.29123009162,0.0,0.0,0.45160600000000006 bbann,BBANN,text2image-1B,10,1592.557788590405,0.0,1000000.0,29064836.0,18250.412140915585,0.0,0.0,0.49080700000000005 bbann,BBANN,text2image-1B,10,2197.8065625321065,0.0,1000000.0,29064836.0,13224.474116827745,0.0,0.0,0.42842900000000006 bbann,BBANN,text2image-1B,10,1540.622672933968,0.0,1000000.0,29064836.0,18865.64212679592,0.0,0.0,0.495423 bbann,BBANN,text2image-1B,10,1571.8388035067858,0.0,1000000.0,29064836.0,18490.97753227373,0.0,0.0,0.486085 bbann,BBANN,text2image-1B,10,1581.271710631964,0.0,1000000.0,29064836.0,18380.671585140848,0.0,0.0,0.49324500000000004 bbann,BBANN,text2image-1B,10,1608.3394385279835,0.0,1000000.0,29064836.0,18071.332023419945,0.0,0.0,0.484772 bbann,BBANN,text2image-1B,10,1555.2814633486682,0.0,1000000.0,29064836.0,18687.830264125092,0.0,0.0,0.487202 bbann,BBANN,text2image-1B,10,2314.605123985524,0.0,1000000.0,29064836.0,12557.146659190483,0.0,0.0,0.425323 bbann,BBANN,text2image-1B,10,1685.0312317828493,0.0,1000000.0,29064836.0,17248.841120439007,0.0,0.0,0.471499.

@PwzXxm Based on the results I pasted above on this thread, and applying the 1500QPS cut off, I see the following results for the 3 datasets you submitted (Without ties...

Could you please rebase from main and squash your commits into 1 so its cleaner to rebase and merge?