python-neo
python-neo copied to clipboard
NeuralEnsemble
An OSError about header when importing neuralynx data
Describe the bug
I was trying to import neuralynx data into python using neo. But, I got an OSError.
This is a screenshot of the error.
Also, I noticed that there is a small bug in the line that raise the OSError. I think .format(an, av) is supposed to fill the first {} with an and the second {} with av. However, only the second string after the + calls .format(an, av) in the source code, which is why the error message is weird.
To Reproduce In order to reproduce, just use the two lines of code in the screenshot. The error occurs when reading a file that I can't share publicly. Please get in touch with me and I will share the file privately.
Environment:
- OS: Linux
- Python version: 3.8.13
- Neo version: 0.11.1
@samuelgarcia @JuliaSprenger @apdavison Could you please take a look at this? Thanks!
Hi, thanks for the report. Could you send us a file to debug this ?
@JuliaSprenger @PeterNSteinmetz: would you have time to handle this, you are our best neuralynx expert!
Hi, thanks for the report. Could you send us a file to debug this ?
@JuliaSprenger @PeterNSteinmetz: would you have time to handle this, you are our best neuralynx expert!
@samuelgarcia @JuliaSprenger @PeterNSteinmetz Yes, I can send you a file to debug this. Please provide me with a link so that I can share the file privately.
Gavin, I can take a look at that. Can you send via email to [email protected] ? Alternatively, do you use Google Drive? If so you could share with me there - username is PeterNSteinmetz
Gavin, I can take a look at that. Can you send via email to [email protected] ? Alternatively, do you use Google Drive? If so you could share with me there - username is PeterNSteinmetz
@PeterNSteinmetz I sent the files to you through email and google drive. Please take a look. Thanks!
I now have a fix for this issue. New undocumented header strings for Cheetah v 5.6.0 caused this. I want to add the test data to ephy_testing_data.
So @redagavin, can you please provide permission and desired credit/attribution for including one shortened .ncs file (26.8 kB) and one shortened .ntt file (22.4 kB) from what you sent me to the repository at https://gin.g-node.org/NeuralEnsemble/ephy_testing_data
@samuelgarcia @JuliaSprenger Also will need some help here again placing the data in the ephy_testing_data repository. (Sorry but this whole gin thing is not very clear to me because I usually think in git terms).
I have my own clone of this project and have that checked out and synced and have a master branch on the web site. What do I need to do to make sure that my master is up to date with the project master? I don't see any sort of sync button.
Then I need to create a branch, unlock the Neuralynx folder, add a new sub-folder with two files, lock it and then commit and push to my branch on my repository. Then open a pull request on the project master?
Is there somewhere where this sort of adding things to an already existing clone is laid out?
@samuelgarcia @JuliaSprenger We have no word yet on the use of the sample files for testing. How do you want to proceed?
I could just create a code branch for Neo with the fixes but without any tests.
Alternately, I could hack the headers on these a bit to anonymize them and either remove all the data records or replace the data with random samples. The test files would then be completely synthetic, but not as realistic.
@samuelgarcia @JuliaSprenger Bumping this as I was just going to add something to NeuralynxRawIO and want to be up to date with the main branch.
I have my own clone of this project and have that checked out and synced and have a master branch on the web site. What do I need to do to make sure that my master is up to date with the project master? I don't see any sort of sync button.
If you are using the gin-cli simply gin download will update your local copy of the branch you have currently checked out.
Then I need to create a branch, unlock the Neuralynx folder, add a new sub-folder with two files, lock it and then commit and push to my branch on my repository. Then open a pull request on the project master?
For adding new files you don't need to unlock existing files (folders can not be in a locked state, only files can). Also note that since git annexed files are not shared between forks, you should push your feature branch directly to the Neuralensemble/ephy_testing_data repo and not your fork. You should have permission to do so as you are already listed as a collaborator.
To be sure the gin-cli runs smoothly ensure that you are using a git-annex<9 and not the latest version, since we noticed some issues with that version (see also this issue)
Tell me if you have further question following the contribution guideline
@JuliaSprenger Actually there is the issue noted here in the Jan 23 comment. We don't have permission to use the test files which were given.
@redagavin Could you comment on this?
So @redagavin, can you please provide permission and desired credit/attribution for including one shortened .ncs file (26.8 kB) and one shortened .ntt file (22.4 kB) from what you sent me to the repository at https://gin.g-node.org/NeuralEnsemble/ephy_testing_data
Well see if that produces anything. I have been in touch with him via email to get the files and wrote and received nothing.
Hi, I solved a similar bug in my fork. The issue was that neo didn't recognize the "RawDataFile" option in the headerstring. This is set when ".ncs" files are created by offline replay of raw data ".nrd" files. You'll find the fork here https://github.com/GeezleCode/python-neo. The branch containing the few changes is called "NeuralynxIORawDataSystem". I haven't created a pull request yet.
Hi @GeezleCode Thanks for fixing another issue of the NeuralynxIO. If you want your fix to be included in neo, could you open a PR?
@redagavin Is this resolved for you? Could you comment on https://github.com/NeuralEnsemble/python-neo/issues/1202#issuecomment-1466565212 ?
@JuliaSprenger I have the code which fixes this issue but the test data has not been added as we don't have permission. I can add it without test data if you prefer.
From my look at #1234, I think that will not resolve this issue as this issue has to do with date formatting being version sensitive.
@PeterNSteinmetz If it's just a small change we could add it anyway. Without test files we can just not guarantee that it will not reoccur at some point in the future...