pyroscope
pyroscope copied to clipboard
grafana
Maximum resolution/frequency of profiles supported
As part of the discussion in https://github.com/pyroscope-io/pyroscope-rs/pull/21 , it would be great to understand the limitations of the frequency that a profile can be posted by an agent to the backend. The granularity does matter for some usecases.
There are a couple of challenges around changing the 10-second upload frequency.
First, you have to change this upload interval on the client side. This should be pretty straightforward, and if the change is not too drastic (e.g from 10 to 1 second) I think you shouldn't notice any differences. However, once you go into higher frequencies I bet at some point you might have to deal with issues of overhead of handling each of these second session being too high (right now each 10 second session is it's own http request), and so this current design might now work very well for this kind of usecase. This is speculative, as I haven't tried doing this, but intuitively I would expect the overhead to be more pronounced as you increase frequency.
After you do that you'll have to deal with the same frequency being hardcoded on the server side. On the storage side all the data that's coming in is split into 10-second buckets, and so even if you reduce profiling session duration on the client side, when it makes it to the server it will be mixed together with other data that fits within that 10-second bucket. It would be relatively easy to change this value in the source code. It is hardcoded unfortunately (that's intentional btw, making this configurable would make the code quite a bit more complex). You'll need to make a custom build of pyroscope. I'm speculating again here that you might see overhead associated with the fact that pyroscope server would have to handle more requests than before (with current design each profiling session is a separate http request)
So to summarize, adjusting this upload frequency is possible, but not trivial. You'd have to modify source code for the server and you'd have to build a custom version of pyroscope server. You may or may not run into performance issues associated with such change.
Can you tell us more about your usecase? We don't often hear from users about this frequency thing being an issue so I'm curious where you're coming from and what kind of problems you're hoping to solve with Pyroscope.
@petethepig thank you for the reply. Could you point us to changes needed on the server side? It seems more than just one constant change.
if the change is not too drastic (e.g from 10 to 1 second)
We're actually thinking of 1-2 seconds :)
I'm curious where you're coming from and what kind of problems you're hoping to solve with Pyroscope.
We want to profile a critical section which spans 2 seconds. And to do it continuously to track perf regressions in the future as well to see how it performs under real world workload.
After you do that you'll have to deal with the same frequency being hardcoded on the server side.
@petethepig I'm wondering, how it works with eBPF agent. It has a 15s collect_interval according to its docs.
Is it possible to provide an explanation why this issue is closed? It is because it is implemented? Was the 10s frequency on the server side made configurable somehow? Or is it because it's unlikely to ever be implemented?
