pyroscope icon indicating copy to clipboard operation
pyroscope copied to clipboard

Query more efficently by using the information about (deduplicated) blocks

Open simonswine opened this issue 2 years ago • 0 comments

This issue is an umbrella issue to use the physical block plan for all query endpoints.

The queryStoreAfter flag should be removed once the work is complete and instead a plan of which blocks to query from which store gateways should be used.

That has been created in PR https://github.com/grafana/pyroscope/pull/2586 and makes sure to:

  • Only query on block once from one of the store-gateways (typically the replication factor there is 3) and once the compactor has deduplicated all blocks in the time window, bypass deduplication.

https://github.com/grafana/pyroscope/pull/2586

- [x] MergeStacktraces according to block plan https://github.com/grafana/pyroscope/pull/2586
- [x] MergeStacktraces while ignoring the labels
- [ ] Use the block plan for the SelectSeries endpoint
- [ ] Use the block plan for the LabelName/LabelValues/ProfileTypes endpoint
- [x] Use the block plan for the MergePprof endpoint
- [x] Use the block plan for the MergeByLabels endpoint
- [ ] Remove queryStoreAfter
- [ ] Move creation of blockPlan to query frontend and split according to the boundaries discovered by block plan
- [ ] Reduce ingester local store

simonswine avatar Nov 13 '23 18:11 simonswine