spikeinterface icon indicating copy to clipboard operation
spikeinterface copied to clipboard

Only assign memmap within boundaries for `write_binary`

Open h-mayorquin opened this issue 9 months ago • 3 comments

This is like #1781 but for writing. I am writing the tutorial for how to use the job_kwargs and the memory heap reports crazy allocations for the following code:


from pathlib import Path
from spikeinterface.core import write_binary_recording
from spikeinterface.core import generate_recording



minutes = 30
recording = generate_recording(num_channels=384, durations=[60.0 * minutes], seed=0)


job_kwargs = dict(n_jobs=1, total_memory="1G", progress_bar=False, verbose=False)
file_path = Path(__file__).parent / "recording_test_n_jobs.dat"
write_binary_recording(recording=recording, file_paths=[file_path], **job_kwargs)

image

Reason being that ta memmap for the whole array (80 GiB) is reserved even if it not used. A possible consequence of this massive heap allocation is that maybe the system will overswap. With the current PR this heap allocations disappears.

image

This will make my life easier for tracking the efficiency of processing and I think is a small increase in complexity.

h-mayorquin avatar May 02 '24 23:05 h-mayorquin