NeuralEnsemble
request: RawIO for Bonsai data
It would be great to have a reader for Bonsai. They store their data in raw binary, so it should be just a matter of parsing the data type, shape, etc. and creating a memmap.
Hi @bendichter , I can have a look into this. Is this example dataset the format you are looking for? I did not yet find more detailed specifications on the format, e.g. what data type to expect from a bonsai sink. Maybe @glopesdev could point to the corresponding documentation?
@JuliaSprenger thanks! I don't know what data would be best. I'd defer to @glopesdev
Hey @JuliaSprenger @bendichter I've been looking more into this using the example dataset @JuliaSprenger found as well as some others like from https://groups.google.com/g/bonsai-users/c/ZdHnXKxgnA8 as well as personally playing around with really simple input/output workflows in Bonsai attached below.
I did not yet find more detailed specifications on the format, e.g. what data type to expect from a bonsai sink.
From what I can tell in these workflows, it all comes down to the MatrixWriter node that writes the simple int16 blob to whatever filename+suffix the user specifies.
While that's simple enough in principle, the issue I'm running into seems to be the the number of channels being parsed by the reader nodes in each of these workflows is not a detail included in the .bonsai header itself (any later nodes regarding channels, like the 'Select Channel' node, seem only to allow or specify subsets of the available channels). The closest I've come to finding such a specification is in the MatrixReader node of my example where such information must be manually specified by the user.
So a neo or spikeinterface reader for this seems like it would have to take num_channels as a required input, though it should in principle be capable of always automatically detected the sampling frequency from the Rhdboard or other reader node.
Maybe @glopesdev can confirm for us if there is any other way to get the shape information required to read the binary blob?
Bonsai_simple_matrix_example.zip
@JuliaSprenger Regarding some other details, since I noticed they allow specification of Row vs. Column major (unclear from 'layout' if this is just on-disk convention or actual data transposition), so here's another example with that option flipped. In both cases, np.memmap(filename="E:/Bonsai/example_output_row.raw", shape=(10,80), dtype="int16", order="C") returned the same traces as from the ground truth NeuroScopeRecording traces.
Bonsa_simple_matrix_example_row.zip
Just following up on this. I know nothing about bonsai, but does this seem feasible or would it fit better with just a binary reader? Is there still interest in this?
Bonsai is a pipeline management system
While it has some capability for having nodes in that pipeline that interact with disk via save/load operations, it is not in and of itself a data format and the basic export capability is a simple, classic binary file (for which the existing binary readers work, as long as you know the right shape/type information)
The hope was that the Bonsai header (which defines the pipeline) might have some of that information (such as number of channels) but in the examples I found it did not and even had to be specified manually in their own GUI
Also, Bonsai is very extensible, so anyone could write their own custom export node to save the data to whatever format they wish, again pointing to how this isn't really something a universal API could guarantee support for
Long term, the best way to interact with the format would be to write exporter nodes that could be loaded into the pipeline itself. But I don't think that's neo's resposbility
Cool. Thanks for the explainer @CodyCBakerPhD. I think we close this and we can point people to binary reader in the case someone wants to read from a node.