test-infra
test-infra copied to clipboard
prometheus
Exercise Remote Write as part of Prombench
It would be valuable to be able to benchmark proposed remote write changes in a reproducible manner. Right now we are dependent on having a few people deploy branches to their instances to see if improvements work for them.
Perhaps have this optional so that remote write is not always exercised?
Shouldn't be difficult to implement.
Are you interested in working on this?
We can enable prombench to setup prometheus in custom ways, if we allow specifying the features to add to both of the test prometheus instances.
Example comments:
/benchmark [remote_write,]
/benchmark master [remote_write,]
/benchmark v2.11.1 [remote_write, remote_read, something_else]
A regex like this would work:
(?mi)^/benchmark\s*(master|[0-9]+\.[0-9]+\.[0-9]+\S*)?\s*(?:\[(.*).*\])?\s*$
https://regex101.com/r/8rF6xa/2/
It will give a captured group with remote_write,remote_read,something_else
we can then pass it as an env var (eg.PROMETHEUS_OPTIONS) to the prombench docker image (which uses make) and we can have a Makefile target which deploys custom manifest files based on PROMETHEUS_OPTIONS
cc @csmarchbanks @krasi-georgiev what do you think?
yep that sound like the way to go, but unless someone has the time to work on this the scalability tests are much higher priority
I have interest in working on this, but am not sure when I will have time. If someone else gets to it first I would not be upset.
As far as as implementation goes, that seems reasonable for how to trigger it.
ok ping me or @geekodour for directions if you find the time to work on this.
@csmarchbanks taking a second look on this issue, will there be need of additional queries /tests to view the results better? or just having the prometheus instances use remote write will be good enough?
It would be nice to have a couple more queries, specifically for some of the remote-write specific data including number of shards, and if a queue is backed up at all. Some good queries exist in the prometheus mixin remote write dashboard: https://github.com/prometheus/prometheus/blob/master/documentation/prometheus-mixin/dashboards.libsonnet#L101
Do we have a remote-write 'black-hole' that would accept all samples (and randomly fail)?
@roidelapluie I'll just mention @csmarchbanks riot response here:
https://github.com/prometheus/prometheus/blob/master/documentation/examples/remote_storage/example_write_adapter/server.go
Yes, will add something like that only, thanks for the links :)
Is anyone working on this? I've recently come across the need to do some remote write benchmarks for Prometheus and I recognize the benefit an easy setup like Prombench could provide. I'd be happy to implement it.
