hnswlib icon indicating copy to clipboard operation
hnswlib copied to clipboard

Optimize rebuildIndex memory allocation

Open Axlgrep opened this issue 1 year ago • 0 comments

  1. The current data_level0_memory_ continuous memory block design can result in memory doubling during large index resize, which is very unfriendly to memory and performance(the realloc operation may trigger an expensive data copy action),so we redesigned data_level0_memory_ (using multiple small memory blocks linked together).
  2. In addition, we found some obvious _mm_prefetch out-of-bounds access issues and fixed them (there may be other hidden issues, please check them as well).

#510

Axlgrep avatar Oct 27 '23 07:10 Axlgrep