BUG: IndexError in Phy2

[asheff794]

Describe the issue:

Hi there,

I'm experiencing IndexErrors when I try to open some of my kilosort4 outputs in phy2. This only happens for some of my sessions, and I am unable to find a common thread of which sessions are affected. Spikes are found in all of my sessions, and the contents of the kilosort4 folder appear normal. The phy2 developers thought it could be an issue of spikes occuring before/after the recording time, but this doesn't seem to be the root issue since I have negative spike times in sessions that can be successfully opened. Regardless, they thought it was likely an issue with the kilosort4 output.

I've tried running kilosort4 through both the spikeinterface and kilosort4 api, the problem persists regardless. I use the intan recording system, so I do have to first use spikeinterface to stitch all of my sessions together.

The index number in the error relates to the number of clusters identified. In the example session I'm posting about here, 36 refers to the number of sorted clusters. I haven't identified what the 41 refers to yet.

Reproduce the bug:

phy template-gui params.py

Error message:

(phy2)[kilosort4]$ phy template-gui params.py
08:38:21.107 [E] init:62 An error has occurred (IndexError): index 41 is out of bounds for axis 0 with size 36
Traceback (most recent call last):
File "/gpfs/gibbs/project/jadi/ags72/anaconda/conda_envs/phy2/bin/phy", line 8, in
sys.exit(phycli())
^^^^^^^^
File "/gpfs/gibbs/project/jadi/ags72/anaconda/conda_envs/phy2/lib/python3.11/site-packages/click/core.py", line 1157, in call
return self.main(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/gibbs/project/jadi/ags72/anaconda/conda_envs/phy2/lib/python3.11/site-packages/click/core.py", line 1078, in main
rv = self.invoke(ctx)
^^^^^^^^^^^^^^^^
File "/gpfs/gibbs/project/jadi/ags72/anaconda/conda_envs/phy2/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/gibbs/project/jadi/ags72/anaconda/conda_envs/phy2/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
return ctx.invoke(self.callback, **ctx.params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/gibbs/project/jadi/ags72/anaconda/conda_envs/phy2/lib/python3.11/site-packages/click/core.py", line 783, in invoke
return __callback(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/gibbs/project/jadi/ags72/anaconda/conda_envs/phy2/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
return f(get_current_context(), *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/gibbs/project/jadi/ags72/anaconda/conda_envs/phy2/lib/python3.11/site-packages/phy/apps/init.py", line 159, in cli_template_gui
template_gui(params_path, **kwargs)
File "/gpfs/gibbs/project/jadi/ags72/anaconda/conda_envs/phy2/lib/python3.11/site-packages/phy/apps/template/gui.py", line 209, in template_gui
model = load_model(params_path)
^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/gibbs/project/jadi/ags72/anaconda/conda_envs/phy2/lib/python3.11/site-packages/phylib/io/model.py", line 1433, in load_model
return TemplateModel(**get_template_params(params_path))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/gibbs/project/jadi/ags72/anaconda/conda_envs/phy2/lib/python3.11/site-packages/phylib/io/model.py", line 339, in init
self._load_data()
File "/gpfs/gibbs/project/jadi/ags72/anaconda/conda_envs/phy2/lib/python3.11/site-packages/phylib/io/model.py", line 419, in _load_data
self.sparse_clusters = self.cluster_waveforms()
^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/gibbs/project/jadi/ags72/anaconda/conda_envs/phy2/lib/python3.11/site-packages/phylib/io/model.py", line 1319, in cluster_waveforms
mean_waveform = self.get_cluster_mean_waveforms(clust, unwhiten=False)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/gibbs/project/jadi/ags72/anaconda/conda_envs/phy2/lib/python3.11/site-packages/phylib/io/model.py", line 1215, in get_cluster_mean_waveforms
template = self.get_template(best_template, unwhiten=unwhiten)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/gibbs/project/jadi/ags72/anaconda/conda_envs/phy2/lib/python3.11/site-packages/phylib/io/model.py", line 959, in get_template
return self._get_template_dense(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/gpfs/gibbs/project/jadi/ags72/anaconda/conda_envs/phy2/lib/python3.11/site-packages/phylib/io/model.py", line 881, in _get_template_dense
template_w = self.sparse_templates.data[template_id, ...]
~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^
File "/gpfs/gibbs/project/jadi/ags72/anaconda/conda_envs/phy2/lib/python3.11/site-packages/numpy/core/memmap.py", line 335, in getitem
res = super().getitem(index)
^^^^^^^^^^^^^^^^^^^^^^^^^^
IndexError: index 41 is out of bounds for axis 0 with size 36

Version information:

Python 3.11, Red Hat Enterprise Linux 8.8 (Ootpa)

Context for the issue:

The output of kilosort4 cannot be manually curated with phy2.

Experiment information:

No response

[asheff794]

Actually finding spike times outside of the recording period was the problem, but in my case it was spikes "occurring" after the recording that were responsible. As suggested by the phy2 developers, running remove_excessive_spikes with spikeinterface solved the issue.

It would be nice if this step of removing spikes outside the recording period is incorporated into the kilosort pipeline, though the priority is lower given there's a functional workaround.

[jacobpennington]

Which version of Kilosort4 was this? Also, just to clarify, you're saying there were spike times that were past the end of the recording?

[asheff794]

'pip info kilosort' just returns Version: 4.0, is there a way to get more specific than that?

Spikeinterface was returning the following message. That message, along with the fact that remove_excess_spikes solved the problem, made me believe spikes past the end of the recording were the issue.

UserWarning: Some spikes exceed the recording's duration! Removing these excess spikes with spikeinterface.curation.remove_excess_spikes() Might be necessary for further postprocessing. warnings.warn(

When I actually look at spike_times.npy I don't see any spike times past the end of the recording. I see some negative spike times that go away once I apply remove_excess_spikes. But I also see negative spike times for sessions that have no issue loading with phy2, so I didn't think those were actually the issue.

[asheff794]

However, it seems the root problem was more anticlimactic. Switching to python 3.9 and re-installing everything fresh solved the issue. I believe it was the python version rather than the fresh install of everything, since I did a fresh install yesterday with python 3.11 and was still getting IndexErrors. Using python 3.9 or 3.10 was clearly stated in the installation instructions so sorry about that!

[jacobpennington]

Great, glad that fixed it! For future reference, conda list kilosort should give more detailed version information (if you're using conda as your environment manager).