NeuralEnsemble
simple proposal for buffer_id in rawio
In your example the idea would be that we move the sync channel into a new stream right? So it will keep the buffer but you'll assign a new stream ?
Hey, Sam, what I had in mind last time we discussed was something like this that I think should be less concentrate the disruption in one PR and then we can work piece by piece on the formats:
https://github.com/NeuralEnsemble/python-neo/pull/1545
In your example the idea would be that we move the sync channel into a new stream right? So it will keep the buffer but you'll assign a new stream ?
In this PR the idea to have no change IOs. Only adding this buffer_id stuff in a very neutral way.
The next PR will have some changes for some IOs (plexon, spikeglx, ced) to split actual stream in more streams but with the same buffer.
Hey, Sam, what I had in mind last time we discussed was something like this that I think should be less concentrate the disruption in one PR and then we can work piece by piece on the formats:
Your proposal is cool and easy but I think I prefer to be brave and explicitly add this features everywhere as a necessary implementation for rawios.
Good. We covered in the discussion today: The buffer will be undefined if:
- The stream is divided across multiple files. Example intan's one-file-per-channel
- There is an API between neo and the data which obfuscates the data layout. Example plexon2
We decide between two approaches:
- Have a strict version where stream and buffer are orthogonal concepts.
- Have a "nested-unless-edge-case" approach where the stream headers contain a reference fo their buffer unless the buffer is hard to define (cases 1 and 2 above).
I am weary of not going with the strict apporach but this will faciliate some work on Sam and I could not find a concrete example where this could cause problems. In such a case "practicallity should beat purity" and we will stick to this.
Good. We covered in the discussion today: The buffer will be undefined if:
I think what I would add to this so we don't forget that we didn't completely settle on a rigid buffer definition so that we have some flexibility as we implement it. Buffers ought to have same dtype, sampling rate, but something like same file vs same memmap was left in the air
- plexon 1 for example is one file, but multiple memmaps.
- Sam mentioned thinking about intan header-attached which is one file, one memmap but with the block structure doesn't quite fit with what he was imagining for a buffer (when you hopped off @h-mayorquin).
Thanks for summary of the discussion @h-mayorquin! I think keeping our documentation up will help us when we go back and think about our choices!
Thanks for this summary of internal discussion @h-mayorquin and @zm711.
Looks like raw binary signal wasn't done quite right (tests failing). Want to fix that? I'll look over this in parallel.
Now plexon2 tests are failing. Heberto has done a bunch of updates. He has another PR that we need to review (but I need to check if it is plexon1 or plexon2). Do we want to get his last PR done and then you can finalize this or do you want to finish this first. There are slightly organizational changes occurring that I think will keep leading to test failures. Up to you both from my perspective.
I think with the latest boost to plexon2 once you fix conflicts and merge from main I hope the plexon2 won't give us any troubles.
@zm711 : it would be nice to merge this soon. So I can continue working on #1513.
scanning through this it looks good. Once tests pass I'll do a final read through, so let's target to merge this by end of weekend. We can ping @h-mayorquin for a read through too, hopefully today if he has time?
If Sam doesn't respond I'll package up your edits into a commit and push them, then once final tests pass we can merge it.
Definitely down for another call at some point to discuss your desire for decoupling more !