Error reading blackrock nev file, IndexError: too many indices, in blackrockrawio.py - due to deprecation drop in Numpy 1.23.0

[histed]

To replicate: brIO = neo.io.BlackrockIO('datafile002.nev')

Error is at:

                    mask = [ev_ids == i]
>                   curr_data = data[mask]
E                   IndexError: too many indices for array: array is 1-dimensional, but 2 were indexed

../../../../miniconda38/envs/mh38a/lib/python3.8/site-packages/neo/rawio/blackrockrawio.py:442: IndexError

Datafile:

datafile002.nev.gz

[histed]

Versions:

(Pdb) neo.version '0.10.0' (Pdb) np.version '1.23.1' (Pdb) sys.version '3.8.10 | packaged by conda-forge | (default, May 10 2021, 22:58:09) \n[Clang 11.1.0 ]'

[cboulay]

I'm encountering this too, using i140703-001-03.nev from https://gin.g-node.org/INT/multielectrode_grasp

The error is in _parse_header.

https://github.com/NeuralEnsemble/python-neo/blob/50abfbce832edb1c0f8364ac34c8a6442e1a3edf/neo/rawio/blackrockrawio.py#L439-L442

data is a structured array (fields for timestamp, packet_id, etc.) of shape (1317,)

mask is a len-1 list containing a boolean array.

If I simply remove the square brackets around mask and use the bool array directly then this fixes the problem: curr_data = data[ev_ids == i]

This is 4-5 year old code, so it's amazing I hadn't encountered this before. I wonder what's different about that file vs the files I was using previously.

[histed]

I am pretty sure my file passed our test suite about a year ago with an older neo version, so I assumed this was due to a neo change, but I didn’t check the neo code.

On Wed, Aug 17, 2022 at 17:42 Chadwick Boulay @.***> wrote:

I'm encountering this too, using i140703-001-03.nev from https://gin.g-node.org/INT/multielectrode_grasp

The error is in _parse_header.

https://github.com/NeuralEnsemble/python-neo/blob/50abfbce832edb1c0f8364ac34c8a6442e1a3edf/neo/rawio/blackrockrawio.py#L439-L442

data is a structured array (fields for timestamp, packet_id, etc.) of shape (1317,)

mask is a len-1 list containing a boolean array.

If I simply remove the square brackets around mask and use the bool array directly then this fixes the problem: curr_data = data[ev_ids == I]

This is 4-5 year old code, so it's amazing I hadn't encountered this before. I wonder what's different about that file vs the files I was using previously.

— Reply to this email directly, view it on GitHub https://github.com/NeuralEnsemble/python-neo/issues/1151#issuecomment-1218516821, or unsubscribe https://github.com/notifications/unsubscribe-auth/AABLCF24BMSLHZ27AZV4FNTVZVMDPANCNFSM56MXGDLQ . You are receiving this because you authored the thread.Message ID: @.***>

[cboulay]

It's in a block of code that only gets touched when there are no .nsx files present. So you can trigger this, for example, by opening a nev file that has been run through a sorter and has an extra -03 on the filename so the matching .nsx files are missed.

The error has been there since at least before Python 3.8, because I tried checking out old commits and nothing before fe56cc3d951c143e800c25f5b10696b31a9cf101 will work with Python >= 3.8, and that particular commit still has the bug.

I'm not terribly interested in tracking it down more precisely than that.

If everyone is happy with my proposed solution above then I can make a PR.

[histed]

Looks to me like the issue is a change in Numpy that changed how indexing works: https://github.com/numpy/numpy/releases

Multidimensional indexing with non-tuple values is not allowed. Previously, code such as arr[ind] where ind = [[0, 1], [0, 1]] produced a FutureWarning and was interpreted as a multidimensional index (i.e., arr[tuple(ind)]). Now this example is treated like an array index over a single dimension (arr[array(ind)]). Multidimensional indexing with anything but a tuple was deprecated in NumPy 1.15.

(https://github.com/numpy/numpy/pull/21029)

So what happened before Numpy 1.23.0 was: mask = [bool_array] was interpreted as mask = tuple([bool_array]) which is equiv to mask = bool_array for this indexing case.

[histed]

Indeed if I change the code in blackrockrawio.py to

             for i in np.unique(ev_ids):
                    mask = tuple([ev_ids == i])
                    curr_data = data[mask]

it works.

Looks like this change was added in 2018 in this commit: https://github.com/NeuralEnsemble/python-neo/commit/780d4c40f9d0443c70382ed863641363448bdbc8

I looked through blackrockrawio.py and it looks like this is the only line with 'mask' in it that has the bool array wrapped into a list. So I vote for dropping the square brackets.

@cboulay I'm good with your solution - please go ahead and make the pull request. Thanks! Mark

[histed]

fyi @muellerbjoern - proposed bugfix