BUG: Tetrode data is garbled in Kilosort4 but not Kilosort3

[afmcelroy]

Describe the issue:

My lab has been using Kilosort3 for spike sorting tetrode data for some time, and recently advanced to Kilosort4. Unfortunately, data that was able to be sorted normally in Kilosort3 doesn't seem to have worked in Kilosort4. Here is a screenshot of the Kilosort4 data from phy. Note the strange shape of the waveforms. Every cluster that can be viewed looks like this: Trying to view certain clusters also yields an out-of-bounds error: Here is the same data (same animal, same recording session) sorted in Kilosort3, looking completely normal: This may be some error with setting up Kilosort4, but I'm not sure what. It could also be an error with phy. Attached is a zipped folder containing channel_map.mat for these data and the python script (main5-1.py) I use to launch Kilosort4. 'n_chan_bin' on line 18 is 48 rather than 64 to exclude broken channels - this was a bug fix that I tried and which didn't work. Another possible issue is 'batch_size' on line 21, which was revised down from its default value because our GPU was running out of memory - our copy of Kilosort3 runs on the department computing cluster, which is less limited, so this may make a difference. Let me know what other files could be needed for debugging. kilosort4 debugging.zip

Reproduce the bug:

No response

Error message:

The text of the out-of-bounds error from phy2:
13:02:31.019 [E] __init__:62          An error has occurred (IndexError): index 1018798 is out of bounds for axis 0 with size 1018768
Traceback (most recent call last):
  File "/home/mouse/miniconda3/envs/phy2/lib/python3.7/site-packages/phy/gui/widgets.py", line 269, in emitJS
    self._debouncer.submit(emit, *args)
  File "/home/mouse/miniconda3/envs/phy2/lib/python3.7/site-packages/phy/gui/qt.py", line 620, in submit
    self.trigger()
  File "/home/mouse/miniconda3/envs/phy2/lib/python3.7/site-packages/phy/gui/qt.py", line 635, in trigger
    f(*args, **kwargs)
  File "/home/mouse/miniconda3/envs/phy2/lib/python3.7/site-packages/phylib/utils/event.py", line 141, in emit
    res.append(f(sender, *args, **kwargs))
  File "/home/mouse/miniconda3/envs/phy2/lib/python3.7/site-packages/phy/cluster/supervisor.py", line 820, in _clusters_selected
    self.similarity_view.reset(cluster_ids)
  File "/home/mouse/miniconda3/envs/phy2/lib/python3.7/site-packages/phy/cluster/supervisor.py", line 379, in reset
    similar = emit('request_similar_clusters', self, cluster_ids[-1])
  File "/home/mouse/miniconda3/envs/phy2/lib/python3.7/site-packages/phylib/utils/event.py", line 141, in emit
    res.append(f(sender, *args, **kwargs))
  File "/home/mouse/miniconda3/envs/phy2/lib/python3.7/site-packages/phy/cluster/supervisor.py", line 737, in _get_similar_clusters
    sim = self.similarity(cluster_id) or []
  File "/home/mouse/miniconda3/envs/phy2/lib/python3.7/site-packages/phy/apps/template/gui.py", line 169, in template_similarity
    temp_i = np.nonzero(self.get_template_counts(cluster_id))[0]
  File "/home/mouse/miniconda3/envs/phy2/lib/python3.7/site-packages/phy/utils/context.py", line 154, in memcached
    out = f(*args, **kwargs)
  File "/home/mouse/miniconda3/envs/phy2/lib/python3.7/site-packages/phy/apps/base.py", line 486, in get_template_counts
    st = self.model.spike_templates[spike_ids]
IndexError: index 1018798 is out of bounds for axis 0 with size 1018768

Version information:

OS: Linux Mint 20.3 Cinnamon version 5.2.7 Python: 3.9.18 (in the conda env for kilosort) Kilosort: 4.0.2 Phy: 2.0b5

Context for the issue:

No response

Experiment information:

64-channel tetrode recording from primary motor cortex of mouse, using Whitematter wireless data logging system

[jacobpennington]

Hello,

Please try sorting this again after installing the latest version of Kilosort (v4.0.4), with the default batch size and dminx = 10. The memory issue is likely related to some bugs that have now been fixed. If you still encounter issues, please let us know.

[afmcelroy]

This seems to have worked, Kilosort4 produced well-sorted units for the original and a second recording after updating, using the default batch size and dminx = 10 - no memory issues or phy errors. Is there any more information you can give me about dminx as a parameter? My understanding is it represents the space in between contacts - is 10 a good value for tetrodes specifically or a general rule of thumb value? I know the default is based on the spacing of Neuropixels contacts.

[afmcelroy]

I did also get this UserWarning: NaNs and/or zeroes present in weights for spikedetect.run, may need to adjust 'min_template_size' and/or 'dminx' for best results. It seems to have not affected the sorting itself but I wanted to note it in case it is a bug or you can give any further context on it.

[jacobpennington]

@afmcelroy The UserWarning you mentioned is not a problem as long as you're getting good sorting results. It's just there as a debugging tool in case other problems pop up (which typically cause the sorting to crash).

dmin and dminx control how the universal templates used in the spike detection step are positioned, and should correspond to the vertical and horizontal spacing, respectively, of contacts on the physical probe. I suggested dminx = 10 because the probe file you shared had contacts spaced 10um apart, more-or-less.

[zmingmin]

@jacobpennington Hi, Jacob, I understand there has been many discussion about dmin and dminx. As shown in the above picture, my understanding is that the default dminx = 32 is used for Neuropixels 2.0, while for Neuropixels 1.0, it should be set to 16. Is this correct?

I am using the Long Column map for a Neuropixels 1.0 recording (as shown below). When I set dminx to 32, the Kilosort processing time is about twice the length of my recording (Which used to be the same time as my recording). However, if I change it to 16, the processing time is much much longer.

Do I need to specify dmin as 20, or are the default settings sufficient?

Thank you!

[jacobpennington]

The default settings should be sufficient.

You should not need to specify dmin. You can set it to 20 if you want, but the default should already do that for you (it gets set to the median difference between unique y values). For the "Bank 0" placement for Neurpixels1, you would still use dminx = 32 since each row still places the contacts 32um apart. For the Long Column placement you're using, dminx = 32 is still appropriate if I'm reading that image correctly. To illustrate why, this is what a small section of the probe looks like for the Bank 0 placement that our default NP1 probe map uses, from the KS4 GUI with "True Aspect Ratio" and "Universal Templates" selected:

For the long column version, it should look something like this instead:

Notice that in both cases, the universal template positions along the horizontal axis are placed exactly on the contacts (green squares), except for some extras in the middle of the probe. Setting dminx = 16 will place additional universal templates at the horizontal midpoints between contacts, which we've found to be unnecessary. This will also increase the sorting time as you found, since there are more templates to process.

As for why it's taking twice as long as the recording time, that depends on your hardware. When you say it used to take less time, are you referring to previous versions of Kilosort? An earlier release of Kilosort4?

[zmingmin]

@jacobpennington Thank you for the explanation! I'm referring to Kilosort2.5 and the initial release of Kilosort4. It's fine as long as the processing time isn't too crazy, probably due to some parameter difference.