EMAworkbench icon indicating copy to clipboard operation
EMAworkbench copied to clipboard

Use `multiprocessing.shared_memory` to reduce pickling overhead in MultiprocessingEvaluator

Open EwoutH opened this issue 1 year ago • 0 comments

I noticed in a profile run of the MultiprocessingEvaluator that pickling takes up a signficant amount of runtime. Since we are using Python 3.8 or later, we can leverage the multiprocessing.shared_memory module to share large data structures between processes without having to pickle them. This can significantly reduce the serialization overhead. We can create a SharedMemory object to store your data, and then use ShareableList or numpy arrays to interact with the shared memory.

EwoutH avatar Apr 03 '23 07:04 EwoutH