big-ann-benchmarks
big-ann-benchmarks copied to clipboard
harsha-simhadri
T1/kst ann t1
codes for Billion-Scale Approximate Nearest Neighbor Search Challenge T1(re-upload) out team name registered for T1 was Kuaishou Technology Billion-Scale ANN Challenge Track1 kst_ann_t1 is our algorithm name
should be ok now
Thanks for addressing the requests. One more thing: could you please add an entry for random-xs dataset to https://github.com/harsha-simhadri/big-ann-benchmarks/blob/main/.github/workflows/benchmarks.yml
@NJU-yasuo I tried to run bigann-1B on F32s_v2 VM. After downloading the index, the load failed with docker exception 137. Could it be that the index is too big for the memory? Same issue with deep-1B and msturing-1B.
@harsha-simhadri I don't have F32_v2 so indexes were evaluated on my own machine and Azure L8sv2 VM and seems work well. I wonder if the index is too big or the query_bs is too big? If the index is not fit in memory, could i re-upload smaller indexes?
@harsha-simhadri I don't have F32_v2 so indexes were evaluated on my own machine and Azure L8sv2 VM and seems work well. I wonder if the index is too big or the query_bs is too big? If the index is not fit in memory, could i re-upload smaller indexes?
I am not sure what happened here. You would have to debug and make sure they run on F32 VM (assuming this is a T1 entry). You may also want to leave a few GB of RAM (out of 64GB) for the OS and other processes and not attempt to push index size close to the limit. Please reach out to me on email if you need help with accessing Azure VMs.
@harsha-simhadri I don't have F32_v2 so indexes were evaluated on my own machine and Azure L8sv2 VM and seems work well. I wonder if the index is too big or the query_bs is too big? If the index is not fit in memory, could i re-upload smaller indexes?
I am not sure what happened here. You would have to debug and make sure they run on F32 VM (assuming this is a T1 entry). You may also want to leave a few GB of RAM (out of 64GB) for the OS and other processes and not attempt to push index size close to the limit. Please reach out to me on email if you need help with accessing Azure VMs.
Thanks, i've already sent an email to your Gmail
@harsha-simhadri hi, could you please check your G-mail about my questions when free
@harsha-simhadri hi ,it seems work well on my Azure Fv32, all experiments done under /mnt
I see the following results for deep-1B. I will try bigann-1B again with your new index. any other datasets?
algorithm,parameters,dataset,count,qps,distcomps,build,indexsize,queriessize,mean_ssd_ios,mean_latency,recall/ap kst_ann_t1,"FaissIVFPQ(nprobe=8,quantizer_efSearch=32)",deep-1B,10,69460.0592205934,9659.5173,1000000.0,59865324.0,861.8668724407197,0,0,0.53951 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=256)",deep-1B,10,9614.604704258865,57403.6261,1000000.0,59865324.0,6226.4987320261,0,0,0.75211 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=64)",deep-1B,10,16391.414553927127,38474.8955,1000000.0,59865324.0,3652.2365902616502,0,0,0.71569 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=64)",deep-1B,10,11275.156238575122,57526.6362,1000000.0,59865324.0,5309.489530192565,0,0,0.7498199999999999 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=128)",deep-1B,10,10596.212267624653,57439.5464,1000000.0,59865324.0,5649.690897841931,0,0,0.7518900000000001 kst_ann_t1,"FaissIVFPQ(nprobe=64,quantizer_efSearch=64)",deep-1B,10,8387.557877753705,76566.8522,1000000.0,59865324.0,7137.396232910729,0,0,0.76973 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=32)",deep-1B,10,17307.739762289606,38570.0483,1000000.0,59865324.0,3458.875903047466,0,0,0.71013 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=96)",deep-1B,10,10906.352580931023,57465.3714,1000000.0,59865324.0,5489.032520796204,0,0,0.75142 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=128)",deep-1B,10,15204.074715687373,38428.8152,1000000.0,59865324.0,3937.4526315785406,0,0,0.7172 kst_ann_t1,"FaissIVFPQ(nprobe=16,quantizer_efSearch=64)",deep-1B,10,36452.96786826431,19283.6208,1000000.0,59865324.0,1642.262002269459,0,0,0.63873
finally it works，bigann is not update yet，please give me a url for upload index for bigann msturing and msspacev
发自我的iPhone
------------------ Original ------------------ From: Harsha Vardhan Simhadri @.> Date: Fri,Nov 12,2021 1:01 PM To: harsha-simhadri/big-ann-benchmarks @.> Cc: NJU-yasuo @.>, Mention @.> Subject: Re: [harsha-simhadri/big-ann-benchmarks] T1/kst ann t1 (PR #69)
I see the following results for deep-1B. I will try bigann-1B again with your new index. any other datasets?
algorithm,parameters,dataset,count,qps,distcomps,build,indexsize,queriessize,mean_ssd_ios,mean_latency,recall/ap kst_ann_t1,"FaissIVFPQ(nprobe=8,quantizer_efSearch=32)",deep-1B,10,69460.0592205934,9659.5173,1000000.0,59865324.0,861.8668724407197,0,0,0.53951 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=256)",deep-1B,10,9614.604704258865,57403.6261,1000000.0,59865324.0,6226.4987320261,0,0,0.75211 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=64)",deep-1B,10,16391.414553927127,38474.8955,1000000.0,59865324.0,3652.2365902616502,0,0,0.71569 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=64)",deep-1B,10,11275.156238575122,57526.6362,1000000.0,59865324.0,5309.489530192565,0,0,0.7498199999999999 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=128)",deep-1B,10,10596.212267624653,57439.5464,1000000.0,59865324.0,5649.690897841931,0,0,0.7518900000000001 kst_ann_t1,"FaissIVFPQ(nprobe=64,quantizer_efSearch=64)",deep-1B,10,8387.557877753705,76566.8522,1000000.0,59865324.0,7137.396232910729,0,0,0.76973 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=32)",deep-1B,10,17307.739762289606,38570.0483,1000000.0,59865324.0,3458.875903047466,0,0,0.71013 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=96)",deep-1B,10,10906.352580931023,57465.3714,1000000.0,59865324.0,5489.032520796204,0,0,0.75142 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=128)",deep-1B,10,15204.074715687373,38428.8152,1000000.0,59865324.0,3937.4526315785406,0,0,0.7172 kst_ann_t1,"FaissIVFPQ(nprobe=16,quantizer_efSearch=64)",deep-1B,10,36452.96786826431,19283.6208,1000000.0,59865324.0,1642.262002269459,0,0,0.63873
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe. Triage notifications on the go with GitHub Mobile for iOS or Android.
@NJU-yasuo Here is the complete set of results I see.
(benchmark) harshasi@f32node2:~/big-ann-benchmarks$ python3 eval/show_operating_points.py --algorithm kst_ann_t1 --threshold 10000 res.csv recall/ap algorithm dataset kst_ann_t1 bigann-1B 0.712190 deep-1B 0.751890 msspacev-1B 0.764542
algorithm,parameters,dataset,count,qps,distcomps,build,indexsize,queriessize,mean_ssd_ios,mean_latency,recall/ap kst_ann_t1,"FaissIVFPQ(nprobe=8,quantizer_efSearch=32)",bigann-1B,10,66612.3092022825,9424.8646,1000000.0,59946616.0,899.933011149025,0,0,0.49187000000000003 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=256)",bigann-1B,10,9205.241823093957,55887.0556,1000000.0,59946616.0,6512.226093790055,0,0,0.7123 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=64)",bigann-1B,10,16249.843576566123,37463.793,1000000.0,59946616.0,3689.058034161568,0,0,0.67381 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=64)",bigann-1B,10,11191.19412379488,55958.145,1000000.0,59946616.0,5356.587986668968,0,0,0.70988 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=128)",bigann-1B,10,10249.61902723161,55901.5736,1000000.0,59946616.0,5848.667725183868,0,0,0.71219 kst_ann_t1,"FaissIVFPQ(nprobe=64,quantizer_efSearch=64)",bigann-1B,10,8320.461754141232,74342.4151,1000000.0,59946616.0,7204.722258372689,0,0,0.72985 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=32)",bigann-1B,10,17439.81019676784,37524.1208,1000000.0,59946616.0,3437.343372642326,0,0,0.66686 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=96)",bigann-1B,10,10644.263731221621,55919.4577,1000000.0,59946616.0,5631.823629488373,0,0,0.71138 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=128)",bigann-1B,10,14854.444783648687,37432.0517,1000000.0,59946616.0,4035.6012542446138,0,0,0.67519 kst_ann_t1,"FaissIVFPQ(nprobe=16,quantizer_efSearch=64)",bigann-1B,10,35605.47846593446,18803.8801,1000000.0,59946616.0,1683.6346141887666,0,0,0.5923700000000001 kst_ann_t1,"FaissIVFPQ(nprobe=8,quantizer_efSearch=32)",deep-1B,10,69460.0592205934,9659.5173,1000000.0,59865324.0,861.8668724407197,0,0,0.53951 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=256)",deep-1B,10,9614.604704258865,57403.6261,1000000.0,59865324.0,6226.4987320261,0,0,0.75211 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=64)",deep-1B,10,16391.414553927127,38474.8955,1000000.0,59865324.0,3652.2365902616502,0,0,0.71569 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=64)",deep-1B,10,11275.156238575122,57526.6362,1000000.0,59865324.0,5309.489530192565,0,0,0.7498199999999999 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=128)",deep-1B,10,10596.212267624653,57439.5464,1000000.0,59865324.0,5649.690897841931,0,0,0.7518900000000001 kst_ann_t1,"FaissIVFPQ(nprobe=64,quantizer_efSearch=64)",deep-1B,10,8387.557877753705,76566.8522,1000000.0,59865324.0,7137.396232910729,0,0,0.76973 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=32)",deep-1B,10,17307.739762289606,38570.0483,1000000.0,59865324.0,3458.875903047466,0,0,0.71013 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=96)",deep-1B,10,10906.352580931023,57465.3714,1000000.0,59865324.0,5489.032520796204,0,0,0.75142 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=128)",deep-1B,10,15204.074715687373,38428.8152,1000000.0,59865324.0,3937.4526315785406,0,0,0.7172 kst_ann_t1,"FaissIVFPQ(nprobe=16,quantizer_efSearch=64)",deep-1B,10,36452.96786826431,19283.6208,1000000.0,59865324.0,1642.262002269459,0,0,0.63873 kst_ann_t1,"FaissIVFPQ(nprobe=8,quantizer_efSearch=32)",msspacev-1B,10,66317.82777350798,11719.36700095511,1000000.0,59671612.0,899.7823662710052,0,0,0.6508561877473051 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=256)",msspacev-1B,10,9281.563000385271,62863.661822895345,1000000.0,59671612.0,6429.0477797245,0,0,0.7704564060581253 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=64)",msspacev-1B,10,15586.713499063988,43059.948935734756,1000000.0,59671612.0,3828.3639462278816,0,0,0.7383749488334016 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=64)",msspacev-1B,10,10753.174420887568,63217.461932050755,1000000.0,59671612.0,5549.208974429962,0,0,0.7523673079547005 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=128)",msspacev-1B,10,10131.539723047565,63006.51719197708,1000000.0,59671612.0,5889.688401878051,0,0,0.7645415472779369 kst_ann_t1,"FaissIVFPQ(nprobe=64,quantizer_efSearch=64)",msspacev-1B,10,8181.474586039804,83086.98437713194,1000000.0,59671612.0,7293.503313182532,0,0,0.7606119525173967 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=32)",msspacev-1B,10,16080.388207300111,43206.205723836814,1000000.0,59671612.0,3710.8315564738987,0,0,0.7161993450675399 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=96)",msspacev-1B,10,10433.456586723076,63084.45442761632,1000000.0,59671612.0,5719.256269866894,0,0,0.7608916632555601 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=128)",msspacev-1B,10,14425.00863011931,42913.43553008596,1000000.0,59671612.0,4136.6777331007015,0,0,0.748891390367035 kst_ann_t1,"FaissIVFPQ(nprobe=16,quantizer_efSearch=64)",msspacev-1B,10,36263.780216396706,22361.335379997272,1000000.0,59671612.0,1645.4879122893929,0,0,0.7068972574703234
@harsha-simhadri that seems ok, what about msturing-1B ?
algorithm dataset kst_ann_t1 bigann-1B 0.712190 deep-1B 0.751890 msspacev-1B 0.764542 msturing-1B 0.756419
kst_ann_t1,"FaissIVFPQ(nprobe=8,quantizer_efSearch=32)",msturing-1B,10,66824.77850163386,10069.2102,1000000.0,59792456.0,894.7647465608596,0,0,0.593676 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=256)",msturing-1B,10,9099.132505939227,54325.28295,1000000.0,59792456.0,6571.225988957957,0,0,0.766613 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=64)",msturing-1B,10,15615.604301120746,37104.68467,1000000.0,59792456.0,3829.0196682115366,0,0,0.713444 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=64)",msturing-1B,10,10939.395109902307,54358.29073,1000000.0,59792456.0,5465.79179189497,0,0,0.7304 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=128)",msturing-1B,10,10290.569001704127,54360.34076,1000000.0,59792456.0,5810.4130092416,0,0,0.756419 kst_ann_t1,"FaissIVFPQ(nprobe=64,quantizer_efSearch=64)",msturing-1B,10,8413.39183312635,71367.53788,1000000.0,59792456.0,7106.81936440628,0,0,0.740756 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=32)",msturing-1B,10,16490.77389595115,37066.83411,1000000.0,59792456.0,3625.812613602104,0,0,0.668101 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=96)",msturing-1B,10,10625.881372482654,54365.84108,1000000.0,59792456.0,5627.058490869445,0,0,0.747961 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=128)",msturing-1B,10,14235.256436820593,37117.2022,1000000.0,59792456.0,4200.30761408289,0,0,0.736908 kst_ann_t1,"FaissIVFPQ(nprobe=16,quantizer_efSearch=64)",msturing-1B,10,35700.37892090531,19349.76173,1000000.0,59792456.0,1674.8409346710584,0,0,0.675843
@harsha-simhadri that seems ok, what about msturing-1B ?
algorithm dataset kst_ann_t1 bigann-1B 0.712190 deep-1B 0.751890 msspacev-1B 0.764542 msturing-1B 0.756419
kst_ann_t1,"FaissIVFPQ(nprobe=8,quantizer_efSearch=32)",msturing-1B,10,66824.77850163386,10069.2102,1000000.0,59792456.0,894.7647465608596,0,0,0.593676 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=256)",msturing-1B,10,9099.132505939227,54325.28295,1000000.0,59792456.0,6571.225988957957,0,0,0.766613 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=64)",msturing-1B,10,15615.604301120746,37104.68467,1000000.0,59792456.0,3829.0196682115366,0,0,0.713444 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=64)",msturing-1B,10,10939.395109902307,54358.29073,1000000.0,59792456.0,5465.79179189497,0,0,0.7304 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=128)",msturing-1B,10,10290.569001704127,54360.34076,1000000.0,59792456.0,5810.4130092416,0,0,0.756419 kst_ann_t1,"FaissIVFPQ(nprobe=64,quantizer_efSearch=64)",msturing-1B,10,8413.39183312635,71367.53788,1000000.0,59792456.0,7106.81936440628,0,0,0.740756 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=32)",msturing-1B,10,16490.77389595115,37066.83411,1000000.0,59792456.0,3625.812613602104,0,0,0.668101 kst_ann_t1,"FaissIVFPQ(nprobe=48,quantizer_efSearch=96)",msturing-1B,10,10625.881372482654,54365.84108,1000000.0,59792456.0,5627.058490869445,0,0,0.747961 kst_ann_t1,"FaissIVFPQ(nprobe=32,quantizer_efSearch=128)",msturing-1B,10,14235.256436820593,37117.2022,1000000.0,59792456.0,4200.30761408289,0,0,0.736908 kst_ann_t1,"FaissIVFPQ(nprobe=16,quantizer_efSearch=64)",msturing-1B,10,35700.37892090531,19349.76173,1000000.0,59792456.0,1674.8409346710584,0,0,0.675843
OK, seems no problem anymore?