dask
Replace custom profiler with `py-spy`
Currently, the custom profiler causes issues on CI (e.g. https://github.com/dask/distributed/runs/7235536612?check_suite_focus=true#step:11:442) and might not produce completely accurate results (#6499). At the same time, there are great profilers which support standard output formats such as py-spy. I suggest that we remove the custom profiler in favor of a py-spy-based alternative. I understand that @gjoseph92 already has some experience with using py-spy to profile distributed (https://github.com/gjoseph92/scheduler-profilers).
If you decide to go this route - let me know if you need anything else from py-spy to make this easier. I'm a huge fan of dask, and use it daily for my work.
I know that ray uses py-spy for its profiling https://github.com/ray-project/ray/blob/886cc4d6747f3ae56b4b0d77534a375494b6457b/doc/source/ray-core/ray-dashboard.rst#profiling-experimental - and I imagine the requirements from dask should be roughly similar.
I have been thinking about providing a python API for py-spy for a while, so that people using py-spy can control this programmatically rather than through the CLI. This should also alleviate some of the issues around requiring sudo - since we could potentially have py-spy run in the process or parent process.
I'm also a big fan of py-spy and would love to see an integration. I guess the way dask and ray approaches profiling is slightly different. In dask we actually have (by default) always a simple profiler running (Extra thread inspecting _current_frames in a loop) and are collecting the output in memory such that we can always visualize the result in a bokeh tab. I think with py-spy this approach would no longer work.
Yeah, if we switched to py-spy, we wouldn't want the profiler to always be on like it is right now. We'd probably want a UI like ray has, where you have to click a button on the dashboard to start (and stop) the profiler.
we could potentially have py-spy run in the process or parent process
I don't realistically think we could use py-spy in dask without this. In https://github.com/gjoseph92/scheduler-profilers#privileges-for-py-spy, I jumped through a few hoops so that py-spy would "just work" in containers (calling prctl(2) automatically in ubuntu to allow ptracing), but I also had to set up the Coiled backed to use an alternative seccomp file for the Docker containers we run to allow the ptrace syscall. That's obviously not something dask can do in deployment environments in general. And for macOS/Windows, it's a non-starter (py-spy on macOS right now simply requires sudo; sneaky seccomp/prctl workarounds don't exist).
However, if py-spy could run within the same process and access memory without process_vm_readv, then it would work anywhere without elevated privileges. I think we'd need that as a baseline, otherwise it'll be really hard to use.
Beyond that part, the other big issue is the dashboard UX. I imagine we'd want to embed speedscope.app in the dashboard (we're certainly not going to make our own thing) and just feed it profiles from py-spy.
One thing we'd probably want from py-spy is the ability to stream incremental output to a file, instead of writing everything at the end. The UX you'd want would be that you can click "start profiling", and then each time you click "refresh", you'd see more of the profile (but you wouldn't lose everything that was recorded before the last refresh). Basically, py-spy would be appending to a file, and each "refresh" would reload the whole file into speedscope. Right now, I don't think py-spy supports this (and the json format wouldn't work for this), but that seems feasible to add. We'd probably also want to be able to set an upper bound on the length of profile recorded (after some number of samples, we'd start dropping old ones off the back?).
Obviously, the best UX would be some sort of live-updating view, but speedscope doesn't support that and I doubt we'd try to add it. https://github.com/benfred/py-spy/pull/91 would be a cool alternative if that got in. I would probably want both this live-updating view and the speedscope view though—being able to see both individual events in speedscope and the time-aggregated flamegraphs is immensely valuable.
So @benfred I think those are the things we'd need:
- Python API—or, really, ability to run py-spy without needing sudo
- Streaming
recordresults to a file, or maybe https://github.com/benfred/py-spy/pull/91
Taking a quick glance at speedscope I think it wouldn't take too much effort to embed it into an individual plot page like the cluster map uses.
We could possibly add some HTTP endpoints for start/stop profiling and one to retrieve the current file, then hook those up to buttons in the individual plot page.
