VectorSimilarity icon indicating copy to clipboard operation
VectorSimilarity copied to clipboard

Develop pre-processor for naive SQ8 [MOD-9238]

Open dor-forer opened this issue 7 months ago • 1 comments

Describe the changes in the pull request

  1. Change the preprocess call to use separate variables for storage_blob_size and query_blob_size, allowing the option of different vector sizes for each (like in the quantization preprocessor).
  2. Added the QuantPreprocessor, which receives a dim in the constructor and calculates the storage_bytes_count. The QuantPreprocessor takes a vector of float32 with size dim, and converts it to a uint8 vector with the following structure: [dim x uint8, min_value, detla] With a storage_bytes_count of: dim * sizeof(uint8_t) + 2 * sizeof(float). In the preprocess function, there are 3 main cases: a. Allocate storage_blob if it was never allocated. b. If storage_blob and query_blob share the same memory, allocate new memory for storage_blob. c. If storage_blob was allocated and they don't share memory, check if enough memory was allocated for the new vector, if not, reallocate.
  3. Added tests to test the following scenarios: QuantizationTest: Tests only one preprocessor of type QuantPreprocessor, checks that the preprocessor works as expected. QuantizationTestWithCosine: Tests QuantPreprocessor after normalization using the CosinePreprocessor. ReallocateVectorQuantizationTest: Tests the scenario of QuantPreprocessor handling memory that was already allocated, and needs to be reallocated. (using DummyStoragePreprocessor to allocate the storage_blob memory withouth allocating the query_blob. ReallocateVectorCosineQuantizationTest: Tests the relocating scenario in the case where query_blob and storage_blob share memory. QuantizationInPlaceTest: Tests the case where enough memory was already allocated, and needs to use the quantization in place.

Which issues this PR fixes 1. MOD-9238

Main objects this PR modified 1.Preprocessors 2. ...

Mark if applicable

  • [ ] This PR introduces API changes
  • [ ] This PR introduces serialization changes

dor-forer avatar May 29 '25 14:05 dor-forer

Codecov Report

Attention: Patch coverage is 74.35897% with 20 lines in your changes missing coverage. Please review.

Project coverage is 96.00%. Comparing base (5ba2efa) to head (a8aee99).

Files with missing lines Patch % Lines
src/VecSim/spaces/computer/preprocessors.h 72.97% 20 Missing :warning:
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #688      +/-   ##
==========================================
- Coverage   96.29%   96.00%   -0.29%     
==========================================
  Files         111      111              
  Lines        6288     6365      +77     
==========================================
+ Hits         6055     6111      +56     
- Misses        233      254      +21     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:
  • :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

codecov[bot] avatar May 29 '25 15:05 codecov[bot]