test-infra icon indicating copy to clipboard operation
test-infra copied to clipboard

prombench: Uneven Load generation for `query_range`

Open bwplotka opened this issue 1 year ago • 1 comments

Despite @bboreham improvements to loadgen (https://github.com/prometheus/test-infra/pull/618) we might need still some love to make the load exactly the same across two instances. For some reason it's uneven, and once uneven situation starts it persists through days:

link1 image

link2 image

This causes allocations, CPU use and HTTP latency metrics uncomparable across instances.

Acceptance Criteria

  • Both instances receives exactly the same load, no matter how much fast they compute it.

bwplotka avatar Dec 12 '23 10:12 bwplotka

Also I am curious if it makes sense to load different things in phases. e.g. 1h one API, 1h second API. Would allow us to isolate issues better.

bwplotka avatar Dec 12 '23 10:12 bwplotka