lance icon indicating copy to clipboard operation
lance copied to clipboard

chore: make vector validation work with fp16

Open chebbyChefNEQ opened this issue 1 year ago • 0 comments

two issues fixed:

  • when using fp16 and cosine together, the distance we get is usually greater than 1e-6 even when we hit an exact match.
    • fixed by comparing the vectors instead of distance
  • fp16 can have edge cases where IVF partition is assigned wrong (very low percentage, fixable by just using more nprobes)
    • fixed by allowing lower validation threshold for fp16

chebbyChefNEQ avatar Feb 15 '24 18:02 chebbyChefNEQ