imp icon indicating copy to clipboard operation
imp copied to clipboard

Add Python pickle buffer support

Open benmwebb opened this issue 1 year ago • 0 comments

Currently when pickling an IMP C++ object we serialize into a C++ std::string and then pass that back to Python as a bytes object, which Python can then pickle. This creates several copies of the underlying data. Consider adding support for pickle protocol version 5 if available (Python >= 3.8, or the backported pickle5 module in Python >= 3.5) as per PEP 574 to store the data in a Python buffer. This should in principle use less memory and be faster, particularly for large objects such as IMP::em::DensityMap or IMP::Model. For example, see the work done for NumPy arrays at numpy/numpy#12011.

benmwebb avatar Mar 30 '23 18:03 benmwebb