[feature request] Dump intermediate flamegraph svg while recording

[ncolomer]

We just used py-spy to profile some ephemeral but long running (hours) jobs. Outcomes was really useful, thanks for this tool!

It would be very handy if we were able to generate intermediate flamegraph SVGs, eg by sending a signal to the py-spy process (as does SIGQUIT on a JVM process for thread dump).

This would have allow us to start analysis of first traces without having to wait for the process being profiled to exit.

[benfred]

I'm working on something that will let you view flamegraph while the process is running. You'll need to install from the latest dev branch, but once you've done that you can use the serve command to start an interactive web server that will let see flamegraphs from the running process, select the time range to view flame graphs off, look at the source code for each function annotated by how long it takes etc:

pip install py-spy==0.4.0.dev1
py-spy serve --pid 12345

# open a browser with http://localhost:8000 to see results

This still need a bunch of work obviously (and I haven't had a chance to work on it recently). I'm thinking about adding an 'export' functionality to this view - where you can select a time range from the running process and dump out a flamegraph you can share with other people.

[ncolomer]

you can use the serve command to start an interactive web server

Sounds very promising! I gonna give it a try

I'm thinking about adding an 'export' functionality to this view - where you can select a time range from the running process and dump out a flamegraph you can share with other people.

Would be useful feature for sure

[boxed]

I have a similar but also different use case in mind. I would like to have py-spy running all the time on all batch and web machines and profiling all python processes, then sending all that data somewhere to be joined into a live view of the entire product.

It seems like the parts could mostly be available for this. I think we might need:

• a way to run py-spy in a mode that it continuously checks for new processes (since our batch jobs are started by cron I don't think the subprocess support is going to work?)
• a way to specify some grouping threshold (like 1 second) and then dumping what we have for that time slot and then starting over
• a way to send those dumps (this seems like it should be outside py-spy and something the end user needs to build or is a separate project)
• a py-spy command to combine some profiles into one profile. This can be used to group slots from many machines and small time slots into bigger chunks, ultimately giving you the possibility to view a profile for your entire product (or even across all products!) for an entire day or week.

I am totally unfamiliar with the code and have written a total of like 20 lines of rust, but from using the tool it sounds like most of the parts for doing all of the above exists in the current code in some form.

[Jongy]

@benfred , did you advance with this idea?

I'm particularly interested in the signal idea - I'm using py-spy in our continuous profiler and we currently invoke it once per some interval (say, 60 seconds) for each target process. I prefer to invoke py-spy once per target process, and simply signal it every interval to dump its existing samples buffer. This is done:

• for performance reasons - avoiding the startup/cleanup overhead of py-spy itself
• for accuracy - avoiding "dead times" between py-spy invocations

Linux's perf has a similar option, --switch-output=signal; on every SIGUSR2 it will write a perf.data.{timestamp} output file with last samples, and empty its buffer.

I'd be happy to take it and add the signal handling capability (on which py-spy shall write the raw collapsed file, or flamegraph, depending on the output format you've asked for)

Btw @boxed , our continuous profiler sounds exactly like what you have in mind :)