Spike2IO chan_info problem

[threefrenchhens]

Apologies in advance - I am new to spike2 and to neo... I have a large number of smr files and would like to write some code to scrape the data all into one place in python.

I am trying to open smr files in python3.7 using neo v0.9.0.dev as follows:

reader = neo.Spike2IO(filename='example1.smr') reader.read()[0]

but get the following error:

chan_info['type'] = dict_kind[chan_info['kind']]
KeyError: 255

I get this error for all smr files I have tried. Could it be a problem with how channels are defined in my files?

I have uploaded two example smr files

Cheers! Dave

[samuelgarcia]

Hi Dave, inded I can reproduc the bug.

Any idea what could be the channel with kind 255 ?

Is there something special in your recording.

What I can do is to bypass the channel with key 255 but maybe you will miss something in your file.

Would you be agree if if put in public acces your file to integrate it in our test suite ?

Samuel

[samuelgarcia]

Hi Dave, the problem with theses file is more complicated. It is not only a prblem of kind=255 with unknown channel type.

When debugging I this stuff for some channel:

OrderedDict([('del_size', 26300), ('next_del_block', 1724344164), ('firstblock', 1720411923), ('lastblock', 1713333970), ('blocks', 26045), ('n_extra', 26143), ('pre_trig', 26202), ('free0', 25909), ('py_sz', 26089), ('max_data', 25953), ('comment', 'eÉd?fLdÍe¢eãe¢eýf$f_gÇf\x17h\x0eg8h\x90g\x94h¯h\x11i'), ('max_chan_time', 1743218820), ('l_chan_dvd', 1733584930), ('phy_chan', 26637), ('title', 'g\x95gÂf×f\x0eg'), ('ideal_rate', 1.3524894e+38), ('kind', 179), ('unused1', 122)])
unknown 179
OrderedDict([('del_size', 28557), ('next_del_block', 1865510711), ('firstblock', 1868394181), ('lastblock', 1862299349), ('blocks', 28232), ('n_extra', 28357), ('pre_trig', 28346), ('free0', 28259), ('py_sz', 28270), ('max_data', 28183), ('comment', "l¦l\x13l½kaj\x01i±fQe2cjaÐ]FZäS\x82I_:£'\x93\x10ëù\x1fæ®ÓQÿ·\x11±Ý®#°i¶\x8aÀ\x10Î\x8fÝ9í\x97ûæ\t\x9e\x16¤ e)õ1"), ('max_chan_time', 1022375909), ('l_chan_dvd', 1153253775), ('phy_chan', 18096), ('title', 'GïH¢I\tJ\x0fJ'), ('ideal_rate', 4924779.5), ('kind', 68), ('unused1', 75)])
unknown 68
OrderedDict([('del_size', 11047), ('next_del_block', -99936540), ('firstblock', -1153902048), ('lastblock', -1920950119), ('blocks', -31900), ('n_extra', -32524), ('pre_trig', -30929), ('free0', -27722), ('py_sz', -23180), ('max_data', -17781), ('comment', 'Ð\rè^þþ\x12¥%\x804(AøJÆQ>WPZS]A_*`\x9c`¬`ó`TaDa>bábÔd\x91fyhïiOl3m\x06o\x93o`q\x13r~t\x05uéuKvÝv'), ('max_chan_time', 1985377780), ('l_chan_dvd', 1961194757), ('phy_chan', 29236), ('title', 'p§kÙdÁ[\x1aO'), ('ideal_rate', 4.327103e-12), ('kind', 58), ('unused1', 24)])
unknown 58
OrderedDict([('del_size', -12149), ('next_del_block', -328540802), ('firstblock', 197786395), ('lastblock', 678173218), ('blocks', 13413), ('n_extra', 16219), ('pre_trig', 18248), ('free0', 20022), ('py_sz', 21634), ('max_data', 22838), ('comment', '[1^Ó_Î_ö]\x87ZÄS`JÓ>[3e)Î"Ä\x1bº\x15k\r\x1c\x05\x06ú\x94îúäÐÝ@ÚdÜ\x01ãqëZ÷W\x054\x13}!Ç.#;UE\x9cO^W}^¯cYh'), ('max_chan_time', 1855941554), ('l_chan_dvd', 1926656363), ('phy_chan', 29675), ('title', 'uêt\x99t\x85rÄo'), ('ideal_rate', 2.0251758e+24), ('kind', 29), ('unused1', 98)])
unknown 29
OrderedDict([('del_size', 17065), ('next_del_block', 1065369961), ('firstblock', 1023098259), ('lastblock', 987184183), ('blocks', 15150), ('n_extra', 15508), ('pre_trig', 15871), ('free0', 16110), ('py_sz', 16734), ('max_data', 17255), ('comment', 'E\x11H_K+N¼P¯RÿT²U5U%T3Q\x10N:J\x9aFÏB>@±?Í>\x86?(A®BÝD_F\x06HKI¼J#K\rL\x06MÛLmMÚMQNWN\xadOçP'), ('max_chan_time', 1393250866), ('l_chan_dvd', 1451971752), ('phy_chan', 22296), ('title', 'Y\x86[Á[\x02]\x01^'), ('ideal_rate', 1.7777399e+18), ('kind', 191), ('unused1', 93)])
unknown 191
OrderedDict([('del_size', 18980), ('next_del_block', 1214073190), ('firstblock', 1215842380), ('lastblock', 1203390589), ('blocks', 18568), ('n_extra', 18400), ('pre_trig', 18611), ('free0', 18790), ('py_sz', 19008), ('max_data', 19203), ('comment', 'LwN{PÛQ T.VÐWÉY\x19[¡[_\\I\\)\\e[äY\x89XYW\x9dTWS_Q·O\x16NåMìKìK>LyL^LLNÃNîO_QõS¨TýVcX'), ('max_chan_time', 1533368723), ('l_chan_dvd', 1586388367), ('phy_chan', 24651), ('title', 'a¤côdþe\x13g'), ('ideal_rate', 2.3839044e+24), ('kind', 1), ('unused1', 105), ('scale', 4.5156527e+25), ('offset', 1.4109014e+26), ('unit', 'jïi,i'), ('interleave', 26772)])
Adc 1

And internal pointer are ouside the file. In short the channel binary header is somehow corrupted. How did you get this file. Is it a true smr generated by offcial spike2 ? Or a file converted by an external tools ?

[threefrenchhens]

Hi Samuel

thanks for looking into this.

I got these files by processing an initial smrx file with raw data using a Spike2 script witten by someone in the lab (who has since left). However, I also get the issue when trying to load the smrx file. Can smrx files be loaded? I have uploaded the original smrx file used to generate the smr example files, as well as its associated s2rx file, to the same directory as before.

I don't believe there is anything in chanel 255.

I am happy for all the files shared to have public access and to be integrated into your test suite.

Thanks! Dave

[samuelgarcia]

Hi Dave, the smrx cannot be loaded. It is not implemented yet in neo. Which tools did the conversion ? Samuel

[threefrenchhens]

The files were converted within Spike2 itself using its scripting language. Perhaps the channel binary header is left over from the smrx file or something...

[IanGMalone]

I get the same error when I try to read a .smrx file from Spike2. The code reader = neo.io.Spike2IO(filename='filename.smrx') produces KeyError: 255. I believe .smrx are 64 bit files and .smr are 32 bit. Hopefully, .smrx support can be added as neo seems like a great tool to help with processing ephys data.

[deeptimittal12]

Is there any update on this issue? I am interested in reading .smrx files from spike2 using neo package but it seems impossible like you mentioned. Is there any other way i could do this? May be using spikeinterface? any suggestions in this direction would be really helpful.

[samuelgarcia]

We could implement the smrx file format but we need the file format specification and a small file for testing. If a spike2 user could contact CED for this and give us the format spec I would be happy to help for the implementation.

[deeptimittal12]

@samuelgarcia Thank you for your response. I have already contacted spike2 ced people they told me that they are developing pyson (matson is already available) library to read .smrx in python though it is work in progress and it will only work on windows. I can provide example .smrx file and the specifications too. Could you please provide me with your email id? Thank you.

[samuelgarcia]

Maybe if pyson is opensource maybe we should be able to infer the specification from the code. But my guess is that pyson will be only a wrapper on top of windows dll. So it will be very retrictive to use (win + closed dll). I think users should insist to have spec.

[deeptimittal12]

http://ced.co.uk/de/downloads/contributed#SONPython This only supports .smr though.

[IanGMalone]

I'm not sure if this helps you, but the workaround that I'm using is to use Spike2 to export the .smrx as a .mat, then read that .mat in Python using the h5py library.

[deeptimittal12]

@IanGMalone I thought about it already. There are many indirect solutions but we would like to minimise these steps.

[samuelgarcia]

@threefrenchhens @deeptimittal12 @IanGMalone : could you check if the fix #824 (already merge in master) fix the problem ? I think I lost theses files.

[threefrenchhens]

Sorry for the slow reply. I still get the same error message... I installed what I believe is the latest merge using:

pip install git+git://github.com/NeuralEnsemble/python-neo

Altgough it still says I have version 0.9.0.dev...

On Thu, 23 Jul 2020, 14:38 Garcia Samuel, [email protected] wrote:

@threefrenchhens https://github.com/threefrenchhens @deeptimittal12 https://github.com/deeptimittal12 @IanGMalone https://github.com/IanGMalone : could you check if the fix #824 https://github.com/NeuralEnsemble/python-neo/pull/824 (already merge in master) fix the problem ? I think I lost theses files.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NeuralEnsemble/python-neo/issues/777#issuecomment-663012173, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFPKEGBO3XY5VJL54SX4WBTR5A4N5ANCNFSM4JV3HFCQ .

[samuelgarcia]

To be sure. Could you try pip uninstall neo and then pip install git+git://github.com/NeuralEnsemble/python-neo ?

[threefrenchhens]

Tried but still get the error...

On Mon, 24 Aug 2020, 10:49 Garcia Samuel, [email protected] wrote:

To be sure. Could you try pip uninstall neo and then pip install git+git://github.com/NeuralEnsemble/python-neo ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/NeuralEnsemble/python-neo/issues/777#issuecomment-679027709, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFPKEGFJ7DTAGPBZ7A4S5DDSCIZQ3ANCNFSM4JV3HFCQ .

[samuelgarcia]

Hi all. We have implemented CedIO (based on sonpy) that read both smrx and smr. Could you try it on your files ?

[htwangtw]

Hi all. We have implemented CedIO (based on sonpy) that read both smrx and smr. Could you try it on your files ?

Not OP, but tested CedIO on smrx file. ~The API seems to be inconsistent with the SpikeIO from 0.9.0?~ (My bad, I was testing CedRawIO rather than CedIO.) The analogue channels are loaded normally but event channels are not loaded. In the test file I have, event channels correspond to sonpy.lib.DataType.EventRise in SonPy API.