lance
lance copied to clipboard
chore: make vector validation work with fp16
two issues fixed:
- when using fp16 and cosine together, the distance we get is usually greater than
1e-6
even when we hit an exact match.- fixed by comparing the vectors instead of distance
- fp16 can have edge cases where IVF partition is assigned wrong (very low percentage, fixable by just using more nprobes)
- fixed by allowing lower validation threshold for fp16