lance icon indicating copy to clipboard operation
lance copied to clipboard

Seed Rng in benchmark

Open chebbyChefNEQ opened this issue 1 year ago • 0 comments

I observed that benchmarks have high variability in latency measurement across runs. (10+%) Increasing the number of samples didn't seem to reduce the variability.

I tried to seed testing data generation with a fixed seed, and the variance in measured latency across runs seems to be much more reasonable. (0-2%)

see this branch: https://github.com/lancedb/lance/compare/main...211805c84e6e

TODO

  • [ ] attach some examples and repro steps

chebbyChefNEQ avatar Jun 01 '23 04:06 chebbyChefNEQ