python-neo icon indicating copy to clipboard operation
python-neo copied to clipboard

`SpikeGLXRawIO` event extraction from `XD` channels is memory inefficient

Open h-mayorquin opened this issue 1 year ago • 3 comments

I am reading a nidq file of about 3 hours with reported size of 2.8 GiB. Reading with the following:

from pathlib import Path 


folder_path = Path("/home/heberto/data/just_nidq")


from neo.rawio.spikeglxrawio import SpikeGLXRawIO


rawio = SpikeGLXRawIO(dirname=folder_path)
rawio.parse_header()

Leads to an RSS allocation of 3.8 GiB.

Command line: /home/heberto/miniconda3/envs/work/bin/memray run --native --follow-fork dev.py
Start time: 2024-12-11 13:47:45.762000-06:00
End time: 2024-12-11 13:47:47.532000-06:00
Duration: 0:00:01.770000
Total number of allocations: 42613
Total number of frames seen: 3262
Peak memory usage: 3.9 GiB
Python allocator: pymalloc

The problem could be solved by delaying this parsing to _get_event_timestamps

https://github.com/NeuralEnsemble/python-neo/blob/0ae6e76d2e2c2d9f9a7b42a23b04f393ee8a11cd/neo/rawio/spikeglxrawio.py#L207-L225

h-mayorquin avatar Dec 11 '24 19:12 h-mayorquin