lance icon indicating copy to clipboard operation
lance copied to clipboard

feat: support remapping vector storage and flat index

Open BubbleCal opened this issue 1 year ago • 2 comments

prepare for supporting remap for new vector index format, HNSW remap not supported because simply mapping the row ids could break the connectivity of graph

BubbleCal avatar Aug 08 '24 11:08 BubbleCal

Codecov Report

Attention: Patch coverage is 77.77778% with 76 lines in your changes missing coverage. Please review.

Project coverage is 79.00%. Comparing base (2b29487) to head (c18b4dc).

Files with missing lines Patch % Lines
rust/lance/src/index/vector/builder.rs 76.29% 1 Missing and 31 partials :warning:
rust/lance-index/src/vector/storage.rs 64.70% 7 Missing and 5 partials :warning:
rust/lance/src/index/vector/ivf/v2.rs 89.15% 8 Missing and 1 partial :warning:
rust/lance/src/index/vector/utils.rs 64.28% 4 Missing and 1 partial :warning:
rust/lance-file/src/v2/writer.rs 69.23% 0 Missing and 4 partials :warning:
rust/lance-index/src/vector.rs 0.00% 3 Missing :warning:
rust/lance-index/src/vector/hnsw/builder.rs 0.00% 2 Missing :warning:
rust/lance-index/src/vector/v3/shuffler.rs 90.90% 0 Missing and 2 partials :warning:
rust/lance/src/dataset/scanner.rs 77.77% 0 Missing and 2 partials :warning:
rust/lance/src/index/vector/ivf.rs 85.71% 1 Missing and 1 partial :warning:
... and 3 more
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #2708      +/-   ##
==========================================
+ Coverage   78.80%   79.00%   +0.19%     
==========================================
  Files         246      246              
  Lines       86637    86900     +263     
  Branches    86637    86900     +263     
==========================================
+ Hits        68278    68655     +377     
+ Misses      15529    15378     -151     
- Partials     2830     2867      +37     
Flag Coverage Δ
unittests 79.00% <77.77%> (+0.19%) :arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov-commenter avatar Aug 08 '24 12:08 codecov-commenter

@BubbleCal I've marked this as draft, since I'm assuming it is not ready for review. (There are no unit tests.) Mark it as ready for review when it is ready.

wjones127 avatar Aug 13 '24 18:08 wjones127