python-neo icon indicating copy to clipboard operation
python-neo copied to clipboard

Fix over-segmentation in BlackRock

Open h-mayorquin opened this issue 2 months ago • 0 comments

This will fix #1770

Testing data is merged here https://gin.g-node.org/NeuralEnsemble/ephy_testing_data/pulls/167 and is used on testing here.

This PR fixes #1770 using the methodology discussed on #1773. To do this, we do three things:

  1. We separate the parsing of the data blocks from the segmentation of the data
  2. I also continue the refactor started in #1772 and #1771 to remove dynamically loaded functions per spec. Now, the data types of the headers and the data blocks are global variables and are separated from the code.
  3. The gaps report (see #1769) was only working for the PTP format. All the formats with timestamps report gaps now.
  4. Following the discussion on #1773 this PR introduces the new gap_tolerance_ms to give user control over the size of the gaps that should create segments. By default, it is None and an error is raised if gaps are found. User can then opt-in to load the data by specifying a gap size.
  5. I used a buffered version of the memmaps so we don't need to create a memamp per block. This reduces the number of memmaps created by the reader which is an OS limit.

This is a large PR but I think that @samuelgarcia prefers them that way rather than sausage sliced PRs and he is the person that might end up reviewing it. I am happy to break it apart if whoever is gona review this prefers it another way.

Tagging @cboulay here as requested in case if he has time to check it.

h-mayorquin avatar Oct 08 '25 03:10 h-mayorquin