lance icon indicating copy to clipboard operation
lance copied to clipboard

refactor vector_index benchmark and add a index building benchmark

Open chebbyChefNEQ opened this issue 1 year ago • 0 comments

Why

We don't have an e2e index building benchmark, which is helpful for understanding latency and profiling for bottlenecks in indexing building.

What changed

  • extract create_file to lance::utils::testing
  • extract out index building from create_file to create_vector_index
  • add a new benchmark vector_index_build, where we bench mark the time required for building Ivf_PQ index

chebbyChefNEQ avatar Jun 01 '23 19:06 chebbyChefNEQ