big-ann-benchmarks first commit

first commit

Open qiaoyuKs opened this issue 2 years ago • 16 comments

T2 competition, a joint team from Kuaishou Technology and Tsinghua University, modify the search process, indexing use the baseline index of the benchmark

Oct 22 '21 15:10 qiaoyuKs

The PR is ready for review and running

Oct 22 '21 17:10 qiaoyuKs

@qiaoyuKs I see the following error when I checkout this PR, install Docker.kota (with success) and try to run msturing-1B. Can you see if you are able to actually run your code by checking out this PR?

(benchmark) harshasi@l8v2node1:~/big-ann-benchmarks$ python3 run.py --algorithm kota-t2 --dataset msturing-1B file data/MSTuringANNS/query100K.fbin already exists file data/MSTuringANNS/query_gt100.bin already exists Traceback (most recent call last): File "run.py", line 6, in main() File "/home/harshasi/big-ann-benchmarks/benchmark/main.py", line 172, in main definitions = get_definitions( File "/home/harshasi/big-ann-benchmarks/benchmark/algorithms/definitions.py", line 138, in get_definitions args = _generate_combinations(run_group["args"]) File "/home/harshasi/big-ann-benchmarks/benchmark/algorithms/definitions.py", line 58, in _generate_combinations l = json.loads(args.strip()) File "/home/harshasi/anaconda3/lib/python3.8/json/init.py", line 357, in loads return _default_decoder.decode(s) File "/home/harshasi/anaconda3/lib/python3.8/json/decoder.py", line 337, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/home/harshasi/anaconda3/lib/python3.8/json/decoder.py", line 353, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Expecting ',' delimiter: line 3 column 3 (char 174)

Oct 23 '21 18:10 harsha-simhadri

We fixed the bug in algo.yml

Oct 24 '21 00:10 qiaoyuKs

@harsha-simhadri Sorry for that , we fixed the problem and now it could run, we test it on azure machine

Oct 25 '21 10:10 qiaoyuKs

@qiaoyuKs I ran your code on Azure L8sv2 VM on the bigann, msturing and msspacev datasets and see the following results on the public query set. Are they as expected?

kota-t2,DiskANN,bigann-1B,10,1729.9645217994996,2268.1108,1000000.0,60379620.0,34902.23021290255,55.7439,4592.0382,0.9371 kota-t2,DiskANN,bigann-1B,10,1313.924291806509,2863.43984,1000000.0,60379620.0,45953.652258749484,74.3488,6001.7167,0.9640000000000001 kota-t2,DiskANN,bigann-1B,10,1157.3305547640223,3206.66504,1000000.0,60379620.0,52171.455900351044,85.1725,6820.1601,0.9731 kota-t2,DiskANN,bigann-1B,10,1420.3258344842309,2677.71058,1000000.0,60379620.0,42511.10451844025,68.5255,5564.5106,0.9581500000000001 kota-t2,DiskANN,bigann-1B,10,1642.9486573844054,2375.40114,1000000.0,60379620.0,36750.76499111366,59.1014,4829.8101,0.94337 kota-t2,DiskANN,bigann-1B,10,2217.5682863981187,1801.02192,1000000.0,60379620.0,27227.85150308561,41.2181,3572.5136,0.89177 kota-t2,DiskANN,bigann-1B,10,1489.4038761039308,2580.0169,1000000.0,60379620.0,40539.45405187512,65.4768,5320.1737,0.9541000000000001 kota-t2,DiskANN,bigann-1B,10,1564.4039965126092,2479.1623200000004,1000000.0,60379620.0,38595.92543524504,62.3378,5068.6112,0.94878 kota-t2,DiskANN,bigann-1B,10,1938.0611173226125,2043.5712,1000000.0,60379620.0,31154.65217289591,48.7581,4092.3886,0.9196 kota-t2,DiskANN,bigann-1B,10,1533.9216597336165,2519.14818,1000000.0,60379620.0,39362.90984409571,63.5898,5166.4773,0.9508599999999999

kota-t2,DiskANN,msturing-1B,10,1869.9105083966315,2199.94934,1000000.0,60658012.0,32438.99198791693,55.69439,4245.34491,0.926499 kota-t2,DiskANN,msturing-1B,10,1708.9818447326425,2377.74471,1000000.0,60658012.0,35493.65500105093,61.39347,4646.6233,0.934886 kota-t2,DiskANN,msturing-1B,10,1829.7687123162966,2242.62236,1000000.0,60658012.0,33150.64444577439,57.06105,4338.58571,0.928661 kota-t2,DiskANN,msturing-1B,10,1606.1513481600634,2508.44176,1000000.0,60658012.0,37766.062376056376,65.60437,4942.56754,0.939817 kota-t2,DiskANN,msturing-1B,10,2151.6371656304373,1945.2767220000003,1000000.0,60658012.0,28191.561741417954,47.59611,3688.2624,0.909615 kota-t2,DiskANN,msturing-1B,10,1840.9913542224767,2228.411372,1000000.0,60658012.0,32948.558862525606,56.60407,4312.21896,0.927989 kota-t2,DiskANN,msturing-1B,10,1898.1747241126752,2171.368234,1000000.0,60658012.0,31955.968662661086,54.78257,4182.6914,0.9249430000000001 kota-t2,DiskANN,msturing-1B,10,2384.73537024552,1774.407974,1000000.0,60658012.0,25435.950989293615,42.20301,3325.88183,0.892944 kota-t2,DiskANN,msturing-1B,10,1974.2327081034416,2098.642458,1000000.0,60658012.0,30724.85414258559,52.46636,4021.74299,0.92079 kota-t2,DiskANN,msturing-1B,10,1441.516326219669,2758.310464,1000000.0,60658012.0,42079.30974952862,73.68436,5507.78782,0.947376

kota-t2,DiskANN,msspacev-1B,10,1375.2978140778016,2810.365493246009,1000000.0,57428288.0,41756.983405451145,82.98396779915404,5785.338279437849,0.9073441124300723 kota-t2,DiskANN,msspacev-1B,10,1538.8314288346342,2547.252121708282,1000000.0,57428288.0,37319.41454009083,73.47434847864648,5165.493246009005,0.9040012279983627 kota-t2,DiskANN,msspacev-1B,10,1637.764743606222,2414.896950470733,1000000.0,57428288.0,35065.04107149582,68.6947400736799,4853.005628325829,0.9019068085686996 kota-t2,DiskANN,msspacev-1B,10,1894.0193603989806,2130.699065356802,1000000.0,57428288.0,30320.855848011273,58.47274525856188,4193.0943170964665,0.8951801064265247 kota-t2,DiskANN,msspacev-1B,10,1244.4963678400923,3068.2946650293356,1000000.0,57428288.0,46145.80603370557,92.29093327875563,6388.655137126484,0.9099501978441806 kota-t2,DiskANN,msspacev-1B,10,1409.6927009906594,2748.5545094828763,1000000.0,57428288.0,40738.16084856108,80.73986901350798,5638.904489016237,0.906951835175331 kota-t2,DiskANN,msspacev-1B,10,1492.545081361286,2617.2697025515076,1000000.0,57428288.0,38476.75270727644,75.98819757129213,5325.529062627917,0.9050859598853869 kota-t2,DiskANN,msspacev-1B,10,1753.873700033979,2274.3819006685767,1000000.0,57428288.0,32743.685020698696,63.63559148587802,4529.054577705007,0.8993280120070951 kota-t2,DiskANN,msspacev-1B,10,1451.1867197104223,2681.414306180925,1000000.0,57428288.0,39573.327966686156,78.30474826033566,5475.296015827535,0.9058568699686178 kota-t2,DiskANN,msspacev-1B,10,1305.3203479806382,2940.7762586983217,1000000.0,57428288.0,43995.55104526099,87.67921953881839,6087.747373447946,0.9086164551780598

Oct 28 '21 05:10 harsha-simhadri

@qiaoyuKs There seems to be some error in algos.yaml for ssnpp-1b dataset.

$ python3 run.py --algorithm kota-t2 --dataset ssnpp-1B downloading https://dl.fbaipublicfiles.com/billion-scale-ann-benchmarks/FB_ssnpp_public_queries.u8bin -> data/FB_ssnpp/FB_ssnpp_public_queries.u8bin... [0.68 s] downloaded 24.41 MiB / 24.41 MiB at 35.95 MiB/s download finished in 0.68 s, total size 25600008 bytes downloading https://dl.fbaipublicfiles.com/billion-scale-ann-benchmarks/FB_ssnpp_public_queries_1B_GT.rangeres -> data/FB_ssnpp/FB_ssnpp_public_queries_1B_GT.rangeres... [1.04 s] downloaded 59.18 MiB / 59.18 MiB at 57.15 MiB/s download finished in 1.04 s, total size 62054024 bytes Traceback (most recent call last): File "run.py", line 6, in main() File "/home/harshasi/big-ann-benchmarks/benchmark/main.py", line 173, in main args.definitions, dimension, args.dataset, distance, args.count) File "/home/harshasi/big-ann-benchmarks/benchmark/algorithms/definitions.py", line 151, in get_definitions query_args = _generate_combinations(run_group["query-args"]) File "/home/harshasi/big-ann-benchmarks/benchmark/algorithms/definitions.py", line 58, in _generate_combinations l = json.loads(args.strip()) File "/usr/lib/python3.6/json/init.py", line 354, in loads return _default_decoder.decode(s) File "/usr/lib/python3.6/json/decoder.py", line 339, in decode obj, end = self.raw_decode(s, idx=_w(s, 0).end()) File "/usr/lib/python3.6/json/decoder.py", line 355, in raw_decode obj, end = self.scan_once(s, idx) json.decoder.JSONDecodeError: Expecting ',' delimiter: line 4 column 43 (char 172)

Oct 28 '21 10:10 harsha-simhadri

@harsha-simhadri The results of msspacev-1B and msturing is as expected, but the results of our bigann-1B is better than yours, kota-t2,DiskANN,bigann-1B,10,16,40,4,1993.6649399788262,2043.5712,1000000.0,60386524.0,30289.20396254816,48.7581,3978.9845,0.9196 kota-t2,DiskANN,bigann-1B,10,16,80,4,1351.1396813525828,2863.43984,1000000.0,60386524.0,44693.02828819962,74.3488,5842.0951,0.9640000000000001 kota-t2,DiskANN,bigann-1B,10,16,65,4,1532.487737150269,2580.0169,1000000.0,60386524.0,39404.246139216426,65.4768,5167.5979,0.9541000000000001 kota-t2,DiskANN,bigann-1B,10,16,60,4,1609.8440084405902,2479.1623200000004,1000000.0,60386524.0,37510.792153392984,62.3378,4928.4757,0.94878 kota-t2,DiskANN,bigann-1B,10,16,62,4,1580.0696310401734,2519.14818,1000000.0,60386524.0,38217.63472553234,63.5898,5018.3922,0.9508599999999999 kota-t2,DiskANN,bigann-1B,10,16,100,4,1188.5724182884517,3206.66504,1000000.0,60386524.0,50805.92740571652,85.1725,6637.1684,0.9731 kota-t2,DiskANN,bigann-1B,10,16,55,4,1690.841651981719,2375.40114,1000000.0,60386524.0,35713.88481542616,59.1014,4695.4606,0.94337 kota-t2,DiskANN,bigann-1B,10,16,70,4,1462.5567963721735,2677.71058,1000000.0,60386524.0,41288.32750275879,68.5255,5406.8211,0.9581500000000001 kota-t2,DiskANN,bigann-1B,10,16,30,4,2283.285925286544,1801.02192,1000000.0,60386524.0,26447.201960666276,41.2181,3468.8617,0.89177 kota-t2,DiskANN,bigann-1B,10,16,50,4,1780.0820119495365,2268.1108,1000000.0,60386524.0,33923.45048971367,55.7439,4458.5932,0.9371

Oct 29 '21 03:10 qiaoyuKs

@harsha-simhadri Sorry, we fixed the problem in ssnpp-1B query set ,it's a problem of comma, the ssnpp-1B will result in a better result, about 0.22, 0.06 increase than the benchmark

Oct 29 '21 03:10 qiaoyuKs

@qiaoyuKs The run for ssnpp-1B still fails. Could you please look into this. The error I see is

2021-11-07 08:57:53,426 - annb.f1c196f80c - INFO - Reading bin file /home/app/data/indices/T2/DiskANN/SSNPPDataset-1000000000/R100_L100_B60_M100/R100_L100_B60_M100_pq_pivots.bin ... 2021-11-07 08:57:53,427 - annb.f1c196f80c - INFO - Metadata: #pts = 2712002176, #dims = 32643... 2021-11-07 08:57:53,427 - annb.f1c196f80c - INFO - Error. File size mismatch. Actual size is 18446744073709551615 while expected size is 354111548124680 npts = 2712002176 dim = 32643 size of <T>= 4 2021-11-07 08:57:53,446 - annb.f1c196f80c - INFO - Traceback (most recent call last): 2021-11-07 08:57:53,446 - annb.f1c196f80c - INFO - File "run_algorithm.py", line 3, in 2021-11-07 08:57:53,447 - annb.f1c196f80c - INFO - run_from_cmdline() 2021-11-07 08:57:53,447 - annb.f1c196f80c - INFO - File "/home/app/benchmark/runner.py", line 228, in run_from_cmdline 2021-11-07 08:57:53,447 - annb.f1c196f80c - INFO - args.upload_index, args.download_index, args.blob_prefix, args.sas_string) 2021-11-07 08:57:53,448 - annb.f1c196f80c - INFO - File "/home/app/benchmark/runner.py", line 91, in run 2021-11-07 08:57:53,448 - annb.f1c196f80c - INFO - elif rebuild or not algo.load_index(dataset): 2021-11-07 08:57:53,449 - annb.f1c196f80c - INFO - File "/home/app/benchmark/algorithms/kota-t2.py", line 221, in load_index 2021-11-07 08:57:53,449 - annb.f1c196f80c - INFO - if (self.index.load_index(index_path, diskannpy.omp_get_max_threads(), num_nodes_to_cache, self.cache_mechanism, False, True, 4, "int") == 0): 2021-11-07 08:57:53,450 - annb.f1c196f80c - INFO - RuntimeError: Caught an unknown exception! 2021-11-07 08:57:56,061 - annb.f1c196f80c - ERROR - ['euclidean', {'R': 100, 'L': 100, 'B': 60, 'M': 100, 'C': 500000, 'CM': 2, 'url': 'https://comp21storage.blob.core.windows.net/publiccontainer/comp21/diskann-T2-baseline-indices/fbssnpp-1B/R100_L100_B60_M100'}] Trying to instantiate benchmark.algorithms.kota-t2.KoTa(['euclidean', {'R': 100, 'L': 100, 'B': 60, 'M': 100, 'C': 500000, 'CM': 2, 'url': 'https://comp21storage.blob.core.windows.net/publiccontainer/comp21/diskann-T2-baseline-indices/fbssnpp-1B/R100_L100_B60_M100'}]) 0 Running kota-t2 on ssnpp-1B Got 100000 queries dist_cmp function for _u8 uses slow implementation. Please contact [email protected] if you need an AVX/AVX2 implementation. Using AVX2 dist_cmp_float function. Loading index Reading bin file /home/app/data/indices/T2/DiskANN/SSNPPDataset-1000000000/R100_L100_B60_M100/R100_L100_B60_M100_pq_compressed.bin ... Metadata: #pts = 1000000000, #dims = 64... tcmalloc: large alloc 64000000000 bytes == 0x34fc000 @ 0x7f83a1a71887 0x7f83a2aca45f 0x7f83a2ad9c93 0x7f83a29c1e77 0x7f83a29f552f 0x7f83a29ce27e 0x566bec 0x50a533 0x50bf44 0x5096c8 0x50a3fd 0x50bf44 0x507cd4 0x509a00 0x50a3fd 0x50bf44 0x506044 0x50a3fd 0x50bf44 0x507cd4 0x50ae13 0x635262 0x635317 0x638acf 0x639671 0x4b0e40 0x7f83be5bfbf7 0x5b2f0a Reading bin file /home/app/data/indices/T2/DiskANN/SSNPPDataset-1000000000/R100_L100_B60_M100/R100_L100_B60_M100_pq_pivots.bin ... Metadata: #pts = 2712002176, #dims = 32643... Error. File size mismatch. Actual size is 18446744073709551615 while expected size is 354111548124680 npts = 2712002176 dim = 32643 size of <T>= 4 Traceback (most recent call last): File "run_algorithm.py", line 3, in run_from_cmdline() File "/home/app/benchmark/runner.py", line 228, in run_from_cmdline args.upload_index, args.download_index, args.blob_prefix, args.sas_string) File "/home/app/benchmark/runner.py", line 91, in run elif rebuild or not algo.load_index(dataset): File "/home/app/benchmark/algorithms/kota-t2.py", line 221, in load_index if (self.index.load_index(index_path, diskannpy.omp_get_max_threads(), num_nodes_to_cache, self.cache_mechanism, False, True, 4, "int") == 0): RuntimeError: Caught an unknown exception!

Nov 07 '21 09:11 harsha-simhadri

@harsha-simhadri We checked the code and it could run in our azure machine, could you provide more information about the indices file size, after we download the indices from the url "https://comp21storage.blob.core.windows.net/publiccontainer/comp21/diskann-T2-baseline-indices/fbssnpp-1B/R100_L100_B60_M100", we got the following

-rw-r--r-- 1 root root 682666672128 Nov 8 02:44 R100_L100_B60_M100_disk.index -rw-r--r-- 1 root root 64000000008 Nov 8 02:17 R100_L100_B60_M100_pq_compressed.bin -rw-r--r-- 1 root root 599264 Nov 8 02:15 R100_L100_B60_M100_sample_ids.bin -rw-r--r-- 1 root root 38352392 Nov 8 02:15 R100_L100_B60_M100_sample_data.bin -rw-r--r-- 1 root root 1032 Nov 8 02:15 R100_L100_B60_M100_pq_pivots.bin_rearrangement_perm.bin -rw-r--r-- 1 root root 268 Nov 8 02:15 R100_L100_B60_M100_pq_pivots.bin_chunk_offsets.bin -rw-r--r-- 1 root root 1032 Nov 8 02:15 R100_L100_B60_M100_pq_pivots.bin_centroid.bin -rw-r--r-- 1 root root 262152 Nov 8 02:15 R100_L100_B60_M100_pq_pivots.bin

did you get the same file?

Nov 08 '21 03:11 qiaoyuKs

@harsha-simhadri could you provide the file size of R100_L100_B60_M100_pq_pivots.bin you used?

Nov 12 '21 07:11 qiaoyuKs

@qiaoyuKs size below -rw-r--r-- 1 root root 262152 Nov 13 14:47 data/indices/T2/DiskANN/SSNPPDataset-1000000000/R100_L100_B60_M100/R100_L100_B60_M100_pq_pivots.bin

Also, I am able to load and run the first 4 query args for ssnpp-1B, but the docker raises exception 137 after that. Your memory requirement might be running too close to the limit. You might want to recalibrate your query args to leave more memory for OS and other background processes

Load index success. Loaded existing index Index memory footprint: 63001892.0 Starting query Running query argument group 1 of 10... Run 1/5... Run 2/5... Run 3/5... Run 4/5... Run 5/5... Running query argument group 2 of 10... Run 1/5... Run 2/5... Run 3/5... Run 4/5... Run 5/5... Running query argument group 3 of 10... Run 1/5... Run 2/5... Run 3/5... Run 4/5... Run 5/5... Running query argument group 4 of 10... Run 1/5... Run 2/5... Run 3/5... Run 4/5... Run 5/5... Running query argument group 5 of 10... Run 1/5...

2021-11-13 15:41:35,188 - annb.9ecde6d6d9 - ERROR - Child process for container 9ecde6d6d9 raised exception 137

For the 4 query args that completed, I see the following results.

kota-t2,DiskANN,ssnpp-1B,60000,2237.9455403144407,1572.170728,1000000.0,63001892.0,28151.664490972365,31.02686,2363.41259,0.1557059316298228 kota-t2,DiskANN,ssnpp-1B,60000,2329.023940719631,1591.910644,1000000.0,63001892.0,27050.770453022236,30.04728,2410.80347,0.14927481771828133 kota-t2,DiskANN,ssnpp-1B,60000,2746.5274514348266,1328.769594,1000000.0,63001892.0,22938.74469271621,24.35615,1937.67508,0.1332291476357355 kota-t2,DiskANN,ssnpp-1B,60000,1968.80

Nov 13 '21 16:11 harsha-simhadri

@harsha-simhadri Actually we couldn't reproduce the problem in our azure machine before, we have modified our source code of our algorithm to decrease the additional memory we introduced compared with diskann algorithm, you can try that. Another question is that we have used up the Azure Sponsorship, is there any additional code or azure machine we can use?

Nov 15 '21 09:11 qiaoyuKs

Dec 02 '21 05:12 qiaoyuKs

@qiaoyuKs I am able to run only the first 3 query arg combinations of ssnpp-1B before the docker runs out of memory (exception 137). You may want to be conservative with memory usage.

For the query combinations that ran, I see a operating point with AP of 0.182 that's above QPS threshold. We will publish this for the competition.

If you could please rebase your code and squash your changes into one commit, I will get started with merging this PR.

You are welcome to keep improving your algorithm and updating your submission post NeurIPS.

Dec 03 '21 18:12 harsha-simhadri

@harsha-simhadri There are so many conflicts to deal with, could I open a new PR and submit the code?

Dec 06 '21 06:12 qiaoyuKs

big-ann-benchmarks big-ann-benchmarks copied to clipboard

first commit

big-ann-benchmarks
big-ann-benchmarks copied to clipboard