[ENH] SIMD vectorization for distance metrics

[sanketkedia]

Description of changes

Adds SIMD vectorization for euclidean, cosine and inner product for x86, x86_64 and arm. Instruction sets whose support has been added are SSE, AVX and NEON.

Test plan

• [+] Tests pass locally with pytest for python, yarn test for js, cargo test for rust

Documentation Changes

No

[github-actions[bot]]

Reviewer Checklist

Please leverage this checklist to ensure your code review is thorough before approving

Testing, Bugs, Errors, Logs, Documentation

• [ ] Can you think of any use case in which the code does not behave as intended? Have they been tested?
• [ ] Can you think of any inputs or external events that could break the code? Is user input validated and safe? Have they been tested?
• [ ] If appropriate, are there adequate property based tests?
• [ ] If appropriate, are there adequate unit tests?
• [ ] Should any logging, debugging, tracing information be added or removed?
• [ ] Are error messages user-friendly?
• [ ] Have all documentation changes needed been made?
• [ ] Have all non-obvious changes been commented?

System Compatibility

• [ ] Are there any potential impacts on other parts of the system or backward compatibility?
• [ ] Does this change intersect with any items on our roadmap, and if so, is there a plan for fitting them together?

Quality

• [ ] Is this code of a unexpectedly high quality (Readability, Modularity, Intuitiveness)

[sanketkedia]

@codetheweb the main issue with using a third-party is that our distance functions have slightly different definition. For e.g. cosine similarity assumes the vectors are normalized

[vercel[bot]]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
chroma	❌ Failed (Inspect)			May 1, 2024 8:21pm

[sanketkedia]

Verified that there is about 10x perf improvement for both arm and x86_64. Going to merge this unless anyone else has any objections.