erikbern
Optimize the peak memory usage when loading dataset.
I've made a small optimization in data loading process that reduces peak memory usage when handling large datasets. Previously, we were using np.array() to convert large datasets from h5py objects into NumPy arrays. This operation, while straightforward, was causing a spike in memory usage due to the creation of a new array copy, which could lead to OOM errors when working with particularly large datasets. To mitigate this issue, I have replaced np.array() with np.asarray() in the relevant sections of the code. The np.asarray() function, unlike np.array(), will attempt to pass through the input data without creating a new array copy if the input is already a NumPy array. This behavior helps to reduce unnecessary memory allocation and can be particularly effective in scenarios where memory is a constraint.
Thanks @Light-V. Did you observe a change in index building time/search performance with this change? I have made very bad experiences without this cast, but this was a long time ago.
@maumueller Thank you for your suggestion. I will give it a try and post the performance comparison later.
Hi, @maumueller
I have run ann-benchmark on qsg-ngt with these two different ways to load dataset. And here's the search result:
diff result.1 result.2 3c3 < 0: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 3.000) 0.994 9433.409
0: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 3.000) 0.994 9616.273 5c5 < 1: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 0.000) 0.461 40888.565
1: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 0.000) 0.462 41178.560 7c7 < 2: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 1.500) 0.922 19599.222
2: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 1.500) 0.923 19792.332 9c9 < 3: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 2.000) 0.730 24640.807
3: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 2.000) 0.732 24812.362 11c11 < 4: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 2.000) 0.980 13130.141
4: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 2.000) 0.981 13396.878 13c13 < 5: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 1.500) 0.626 28672.532
5: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 1.500) 0.628 28876.686 15c15 < 6: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 5.000) 0.994 7805.384
6: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 5.000) 0.994 7890.528 17c17 < 7: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 1.500) 0.954 16309.917
7: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 1.500) 0.956 16625.485 19c19 < 8: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 3.000) 0.955 14523.425
8: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 3.000) 0.955 14687.466 21c21 < 9: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 0.000) 0.813 27601.332
9: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 0.000) 0.814 27835.597 23c23 < 10: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 8.000) 1.000 2951.642
10: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 8.000) 1.000 3008.100 25c25 < 11: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 3.000) 0.998 7070.455
11: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 3.000) 0.999 7199.444 27c27 < 12: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 1.200) 0.696 28557.780
12: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 1.200) 0.698 28513.283 29c29 < 13: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 1.200) 0.875 22116.188
13: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 1.200) 0.876 22458.581 31c31 < 14: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 3.000) 0.980 11920.713
14: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 3.000) 0.981 12182.230 33c33 < 15: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 5.000) 1.000 4511.494
15: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 5.000) 1.000 4587.789 35c35 < 16: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 2.000) 0.991 10027.577
16: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.04, 2.000) 0.992 10132.040 37c37 < 17: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 2.000) 0.845 21594.956
17: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 2.000) 0.844 21941.602 39c39 < 18: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 1.200) 0.910 18668.998
18: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.02, 1.200) 0.912 19092.833 41c41 < 19: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 1.200) 0.538 31914.098
19: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 1.200) 0.536 32022.208 43c43 < 20: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 0.000) 0.625 36063.251
20: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 0.000) 0.628 36227.625 45c45 < 21: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 1.200) 0.803 25712.051
21: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 1.200) 0.804 25995.405 47c47 < 22: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 10.000) 0.999 4170.250
22: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 10.000) 0.999 4243.505 49c49 < 23: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 2.000) 0.911 19146.093
23: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 2.000) 0.911 19330.184 51c51 < 24: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 0.000) 0.740 32339.626
24: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 0.000) 0.740 32329.880 53c53 < 25: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 2.000) 0.955 16193.192
25: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 2.000) 0.956 16318.249 55c55 < 26: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 3.000) 0.841 19298.415
26: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.9, 3.000) 0.840 19437.719 57c57 < 27: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 20.000) 1.000 2212.972
27: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 1.0, 20.000) 1.000 2235.706 59c59 < 28: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 1.500) 0.771 25701.573
28: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 1.500) 0.772 25698.865 61c61 < 29: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 1.500) 0.860 22676.155
29: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.98, 1.500) 0.861 23154.836 63c63 < 30: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 3.000) 0.916 16764.549
30: QSG-NGT(100, 64, 120, 96, 100, 60, 300, 400, 0, 0.95, 3.000) 0.913 16869.750
And the build time is 3791.798057794571s and 3794.1561596393585s. I think there is no significant different between them.