spikeinterface
spikeinterface copied to clipboard
Only assign memmap within boundaries for `write_binary`
This is like #1781 but for writing. I am writing the tutorial for how to use the job_kwargs and the memory heap reports crazy allocations for the following code:
from pathlib import Path
from spikeinterface.core import write_binary_recording
from spikeinterface.core import generate_recording
minutes = 30
recording = generate_recording(num_channels=384, durations=[60.0 * minutes], seed=0)
job_kwargs = dict(n_jobs=1, total_memory="1G", progress_bar=False, verbose=False)
file_path = Path(__file__).parent / "recording_test_n_jobs.dat"
write_binary_recording(recording=recording, file_paths=[file_path], **job_kwargs)
Reason being that ta memmap for the whole array (80 GiB) is reserved even if it not used. A possible consequence of this massive heap allocation is that maybe the system will overswap. With the current PR this heap allocations disappears.
This will make my life easier for tracking the efficiency of processing and I think is a small increase in complexity.