xorfilter icon indicating copy to clipboard operation
xorfilter copied to clipboard

Do better than "Too many iterations. Are all your keys unique?"

Open lemire opened this issue 2 years ago • 0 comments

Instead of failing with "Too many iterations. Are all your keys unique?", I recommend filling in the fingerprint buffer with 1s.

Example in Go:

		if iterations > MaxIterations {
			// The probability of this happening is lower than the
			// the cosmic-ray probability (i.e., a cosmic ray corrupts your system),
			// but if it happens, we just fill the fingerprint with ones which
			// will flag all possible keys as 'possible', ensuring a correct result.
			for i := 0; i < len(filter.Fingerprints); i++ {
				filter.Fingerprints[i] = ^uint8(0)
			}
			return filter, nil
		}

Reference: https://github.com/FastFilter/xorfilter/blob/6e1eda073dd4927a1c807e2cc47e4a66b56ad7a6/fusefilter.go#L104-L113

lemire avatar Jan 05 '23 15:01 lemire