grafana
The sample count displayed is wrong
Describe the bug
Our currently displayed sample count in the UI is wrong. It seems to represent the sum of values multiplied by 10e9. In order to correct this we need to signal the sampling rate from the collection point (SDK/alloy/agent) and persist it in Pyroscope.
The OG Pyroscope endpoint has this metadata, which is unused by the adapter:
https://github.com/grafana/pyroscope/blob/ff0426278f16120b951cb4466b103349b0097c9d/pkg/og/ingestion/ingestion.go#L50
We then need to calculate the average sample rate for the profiles in a flamegraph (sample rate is a flamegraph wide parameter in flamebearer) and update our read path accordingly.
To Reproduce
Steps to reproduce:
- Query e.g. a CPU profile:
Both the value in the tool tip and and the total value, are wrong and just derived by
Expected behavior
This is the hard part to define the right count of samples in a flameGraph.
For time unit profiles:
- Sum of values / sampleRate, while the sampleRate is only valid per profile
For memory/object unit profiles:
- Unsure what we can do here, go for example scales up the sampled in-use space to represent the actual memory usage
Next steps
Variant A:
- [ ] Update the default value for CPU to be more correct for defaults:
Variant B:
- [ ] Persist sample rate per profile
- [ ] Adapt query path to use avg. sampleRate for profiles in a flamegraph
- [ ] Adapt query path display indivual sample counts per flamegraph box
