pull pprof: allow infrequent profiling

[redbaron]

currently there seem to be no way to have infrequent profiling, because scape interval is added as ?seconds param, making profiling continuous.

It would be nice if it was possible to specify custom seconds option. For a scrape_interval: 5m job I tried to create pprof_config with /debug/pprof/profile?seconds=3s path , but it had 2 problems:

• ? got URLencoded, so it wasn't recognized by pprof server handler
• all path from default parca pprof_config profile config are still present, including /debug/pprof/profile?seconds=300

[brancz]

For cpu profiling I would recommend to always profile using parca agent, it puts way less overhead on the workload so you can always do it.

That said I think this use case does make sense. How about we allow specifying a duration in the scrape config and if specified we use that rather than the scrape interval? Eg. here https://github.com/parca-dev/parca/blob/fc8435ec173dbde62ce48cf4a84d9c7b08090421/parca.yaml#L23

What do you think?

[redbaron]

For cpu profiling I would recommend to always profile using parca agent

Thats what I tried first, but it gave me odd symbol names, unlike native pprof. "shape" of the flamegraph was very similar though. Also agent doesn't seem to be offering off-cpu profiling. Having said that, even with pprof continuous profiling enabled CPU usage graph on a 1 minute resolution chart didn't have any noticeable change in my case.

What do you think?

this and disabling builtin config when pprof_config is provided would do the trick

[brancz]

It’s normal that the agent data looks slightly different. It’s actually good, because 1) the runtime hides some things in the “native” pprof profilers, and 2) it uses sigprof which potentially changes program flow. That said, if it looks suspicious we’d love to see it to be able to fix any bugs should there be any.

this and disabling builtin config when pprof_config is provided would do the trick

Agreed! Do you want to give it a try and implement it?