High memory consumption for long-running tasks

[Andrei-Aksionov]

I am using pyinstrument for profiling some ML/NLU specific tasks and noticed that when running profiling on function with a lot of simple computations inside memory consumption stedealy increases.

For example such a dummy code:

for _ in range(10_000_000):
    res = [x for x in ["qwerty"] * 100]

when running python main.py consumes ~3.2 MB, while pyinstrument main.py - ~40 MB. And the main issue is that while running without profiling memory consumption stays at the same level, with profiling - constantly increases.

[joerick]

Yes, this is not too surprising, as pyinstrument is recording frame samples as the program executes.

If you need to reduce the memory consumption, for a long-running job, you might try reducing the interval a bit. The default is to record every millisecond, but this might be too often for a long running job.

[Andrei-Aksionov]

Thanks for the response.

interval parameter affects memory consumption for long running program quite a bit. I think such information should be reflected in the README file (not as Known issues, but something like good to know). In docs I noticed only this mentioning regarding interval value, but as for me:

but keep in mind that smaller intervals could affect the performance overhead of profiling.

means that the code will be running slower with smaller interval value, not that memory will be consumed like crazy.

In addition, as I understand, it's not possible to set the interval value from CLI.

Plus, it's not a solution, more like a workaround. Memory consumption still increases only now with a slower pace (with bigger interval value). Maybe in some future you are planning to change the code so that it dumps profiling data to disk in chunks in order to not store everything in memory and before preparing output to load all previously dumped data?

[dralley]

Maybe in some future you are planning to change the code so that it dumps profiling data to disk in chunks in order to not store everything in memory and before preparing output to load all previously dumped data?

This would be great. We have some jobs which can take 20 minutes (or longer), and by that point the memory overhead of pyinstrument grows to >20gb. Without pyinstrument it's under 5gb. On some systems this causes out of memory errors.

[joerick]

Thanks for chiming in. I'm curious if reducing the profiling interval would be a solution for your use case, or if you really need millisecond accuracy. The issue I see with dumping intermediate output to disk is that at some point pyinstrument is going to have to load it all into RAM to analyse it and produce reports, which is going to hit the same RAM limits as you're seeing during execution.

[dralley]

I don't suppose there is any way to compress or filter out unnecessary data during the runtime?

[joerick]

Compress, maybe, but we'd still have the issue when analysing. What would you consider unnecessary data?