lance
lance copied to clipboard
Seed Rng in benchmark
I observed that benchmarks have high variability in latency measurement across runs. (10+%) Increasing the number of samples didn't seem to reduce the variability.
I tried to seed testing data generation with a fixed seed, and the variance in measured latency across runs seems to be much more reasonable. (0-2%)
see this branch: https://github.com/lancedb/lance/compare/main...211805c84e6e
TODO
- [ ] attach some examples and repro steps