Vector Similarity Index

[CaptainCodeman]

The field type / index building part of supporting Vector Similarity

Part of #85

This is just the index-building and support for saving / loading objects with binary Buffers. The search part is still remaining to be done although it is possible to use with the redis client directly in case anyone wants to experiment with it.

Also adds support for storing binary data, even if it isn't a vector indexed (e,.g. a profile object with an avatar image, product with photo etc...)

[CaptainCodeman]

This is probably at the point where it is ready for review.

I've successfully used it to store and index both image and text vectors so it's already usable if you're willing to do the search part via the redis client (until we add VSS querying support to this package).

I don't know why the node 14 build started playing up part way through - all the exact same dependencies worked on other branches AFAICT, but I needed to install esbuild directly.

[CaptainCodeman]

Here's some examples of the image matching, the top left image in each set is the source that was searched for and the rest are the matches (out of about 1,500 images - a mix of paintings, photos and digital art). Perfect for a "you might also like ..." product page:

1

2

3

4

5

6

7

8

9

10

11

[codecov-commenter]

Codecov Report

:exclamation: No coverage uploaded for pull request base (main@3c3d8e6). Click here to learn what that means. Patch has no changes to coverable lines.

Additional details and impacted files

@@           Coverage Diff           @@
##             main     #104   +/-   ##
=======================================
  Coverage        ?   98.65%           
=======================================
  Files           ?       35           
  Lines           ?     3047           
  Branches        ?      600           
=======================================
  Hits            ?     3006           
  Misses          ?       41           
  Partials        ?        0           

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

:umbrella: View full report at Codecov.
:loudspeaker: Do you have feedback about the report comment? Let us know in this issue.

[CaptainCodeman]

Finally working to index both HASH and JSON data structures with vector similarity.

The tests can use some refactoring to re-use the common code, but it's usable to build the index and store the data correctly.

[simon-two]

+1 for merging this (I'm not a committer/reviewer, just would like to see it added)

[guyroyse]

No worries. It will be merged. Probably early next year. Like February. Just a matter or timing.

[CaptainCodeman]

I notice vector fields now support FLOAT64 as well as FLOAT32 for the type parameter so that should probably be accounted for.

Ideally it would determine that from the model, whether it had a Float32Array or Float64Array property (instead of Buffer) but that would need reflect-metadata and be more suited to a TypeScript decorator mapping approach which I might have a play around with.

[Shaun-Regenbaum]

+1 for merging, although I do think support for FLOAT64 is important.

[anaibol]

+1 for merging

[simon-two]

Another vote for merging (actually I've voted twice...)

[mauriciocirelli]

Has this been abandoned? This is a very useful feature for redis-om with the popularity of LLM and Embedding documents.

[CaptainCodeman]

Has this been abandoned? This is a very useful feature for redis-om with the popularity of LLM and Embedding documents.

Kinda. It would need some effort to rebase onto the latest version and adapt to wherever the lib has diverged since this was submitted, but I don't personally have time to devote to doing that right now.

I've mostly just been using the node redis client directly or else my own decorator-based mapper / index builder that I've been experimenting with.