xorfilter
xorfilter copied to clipboard
Do better than "Too many iterations. Are all your keys unique?"
Instead of failing with "Too many iterations. Are all your keys unique?", I recommend filling in the fingerprint buffer with 1s.
Example in Go:
if iterations > MaxIterations {
// The probability of this happening is lower than the
// the cosmic-ray probability (i.e., a cosmic ray corrupts your system),
// but if it happens, we just fill the fingerprint with ones which
// will flag all possible keys as 'possible', ensuring a correct result.
for i := 0; i < len(filter.Fingerprints); i++ {
filter.Fingerprints[i] = ^uint8(0)
}
return filter, nil
}
Reference: https://github.com/FastFilter/xorfilter/blob/6e1eda073dd4927a1c807e2cc47e4a66b56ad7a6/fusefilter.go#L104-L113