matrixone icon indicating copy to clipboard operation
matrixone copied to clipboard

[BugFix]: Create `CROSSJOIN_L2` operator

Open arjunsk opened this issue 9 months ago • 0 comments

What type of PR is this?

  • [ ] API-change
  • [x] BUG
  • [ ] Improvement
  • [ ] Documentation
  • [ ] Feature
  • [ ] Test and CI
  • [ ] Code Refactoring

Which issue(s) this PR fixes:

issue # https://github.com/matrixorigin/matrixone/issues/16001

What this PR does / why we need it:

  1. Implement CROSS_JOIN_L2 for mapping tbl embeddings to centroids.
select 
centroids.`__mo_index_centroid_version`, 
centroids.`__mo_index_centroid_id`, 
tbl.pk,
tbl.embedding
from tbl cross_l2 join centroids using(centroids.`__mo_index_centroid`, tbl.embedding);
  1. Optimize memory allocation in NormalizeL2 by using sync.Pool

  2. Modify CREATE INDEX logic to improve the index creation time.

  3. Modify INSERT logic to make use of the new CROSS JOIN L2 operator.

Benchmark

  1. CREATE INDEX
  • VECF32(128) : reduced from 2mins to 29seconds -- good
  • VECF32(960): reduced from 12mins to 3mins 7secs. -- good
  1. QPS
  • VECF32(128) : 55 -- good
  • VECF32(960): 3 -- bad

arjunsk avatar May 16 '24 18:05 arjunsk