filprofiler
filprofiler copied to clipboard
pythonspeed
Track memory from subprocesses
multiprocessing and other similar systems are basically the same program, just with multiple processes. That memory should be tracked together.
Implementation ideas:
- Each subprocess tracks its own info.
- Share critical info with parent (callstack ids and the like).
- Have global counter, perhaps shared via messages, perhaps in shared memory.
- Each process has local counter, and if it's diverged sufficiently from global one, send out message with update.
Indeed, it seems easier to record each process separately and then collate the results when creating the svg.
When I said "record each process separately" I meant "and have separate SVGs", just to be clear. The data cannot be combined if recorded separately, because it's peak memory usage. So peak for process A might be at time T0, peak for process B might be at time T1, but combined peak (which is what is relevant for hardware provisioning) might be time T2.
I am not sure what design I will go for with Fil, yet, but for the production-grade high performance version I'm working on I will probably combine everything (it uses sampling, so there's less data to copy around, so the overhead isn't as bad).
What is your specific use case for multiple process profiling?
My use case is celery workers. I can test using specific tests on the workers but it would be nice to just test the whole system in unison to see if we are getting memory leaks in the way the workers are handled by the processes.
Thanks, that's helpful. For that maybe you just want to look at overall system memory (or maybe process memory) over time using psutil? Will also save you the performance overhead of the open source version of Fil.
This is not a priority at the moment. This will get added to https://pythonspeed.com/sciagraph/ first, and if that turns out to be portable might do it for Fil, but may or may not happen here, on the theory that if one is doing manual develop-time profiling, it's possible to modify code, whereas Sciagraph needs to support unchanged production code.