core
core copied to clipboard
OCR-D
Profile mem
When running processors with ocrd process, we already measure the wall and CPU time of a processor run. This adds basic tracking of memory usage with https://github.com/pythonprofilers/memory_profiler.
Currently, this just outputs a sparkline of memory usage and max/min memory, e.g.
16:35:54.593 INFO ocrd.process.profile - memory consumption: ▁▁▂▅▅▃▄▄▄▄▄▄▄▄▄▄▅▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▄▅▅▅▅▅▅▅▅▅▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆██▅▅▅▅▅▅▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▆▅▅▅▅▆▆▆▆▆▆▆▆▆▆▆▆▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▇▆▄▄
max: 300.79 MiB min: 133.09 MiB
This will conflict with #652 (where that part of run_processor is delegated to the new run_api) – not sure what's the least painful way for merging.
Oh, since I saw this also uses multiprocessing, have you measured any CPU overhead for the memory profiling yet?
This will conflict with #652 (where that part of run_processor is delegated to the new run_api) – not sure what's the least painful way for merging.
Is that just an issue of resolving conflicts or do you mean there's an incompatible API-wise now?
Oh, since I saw this also uses multiprocessing, have you measured any CPU overhead for the memory profiling yet?
I haven't, yet. I did notice that when I profile the current process (proc=-1) instead of the processor.process() call only, then either it just offers a single frame of measurement (max_iter=1) or will hang until the timeout is reached (timeout=100)
This will conflict with #652 (where that part of run_processor is delegated to the new run_api) – not sure what's the least painful way for merging.
Is that just an issue of resolving conflicts or do you mean there's an incompatible API-wise now?
It looks like this is only cosmetic (since the workflow server does not handle the retval either, it just adds exception handling, and does not use multiprocessing itself, at least in the current implementation). But I would have to check/test.
I wonder if it would be much effort to incorporate gputil for (avg/peak) GPU memory usage statistics. (Or perhaps one could add it to memory_profiler as an alternative backend?)
I wonder if it would be much effort to incorporate gputil for (avg/peak) GPU memory usage statistics. (Or perhaps one could add it to memory_profiler as an alternative
backend?)
I'm afraid it's not that simple at all. GPutil only gives snapshots, no aggregation/statistics there. And it looks like one would need to use the respective ML framework's facilities for process/session measurements anyway. (See here and there for Pytorch. TF now also has things like get_memory_info / get_memory_usage / reset_memory_stats.)
Let's treat that off-topic for now!
@kba we could really use that prior to #875. Would you consider fast-tracking this?
@kba we could really use that prior to #875. Would you consider fast-tracking this?
Sure. It will be temporarily inconsistent with CLI flags vs environment variables to configure behaviors and we should definitely continue the discussion on configuration options.