Spyking circus crash during matching

[paul-aparicio]

Hello, I have been managing to crash Spyking circus with the following error. It seems to happen randomly (I have tried running it a number of times, and it will crash with the same error at different probes). I am sorting a Utah array and run the sorter using the run_sorter_by_property function. This recording does have several instances of drop out, could this be the underlying problem? If so, might there be a parameter I could change to help? Thanks for your time!

SpikeSortingError: Spike sorting error trace:
Traceback (most recent call last):
  File "/media/paul/storage/pdev/test/spikeinterface/src/spikeinterface/sorters/basesorter.py", line 270, in run_from_folder
    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)
  File "/media/paul/storage/pdev/test/spikeinterface/src/spikeinterface/sorters/internal/spyking_circus2.py", line 272, in _run_from_folder
    labels, peak_labels = find_cluster_from_peaks(
                          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/paul/storage/pdev/test/spikeinterface/src/spikeinterface/sortingcomponents/clustering/main.py", line 44, in find_cluster_from_peaks
    outputs = method_class.main_function(recording, peaks, params, job_kwargs=job_kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/paul/storage/pdev/test/spikeinterface/src/spikeinterface/sortingcomponents/clustering/random_projections.py", line 182, in main_function
    labels, peak_labels = remove_duplicates_via_matching(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/media/paul/storage/pdev/test/spikeinterface/src/spikeinterface/sortingcomponents/clustering/clustering_tools.py", line 666, in remove_duplicates_via_matching
    similar_templates = detect_mixtures(
                        ^^^^^^^^^^^^^^^^
  File "/media/paul/storage/pdev/test/spikeinterface/src/spikeinterface/sortingcomponents/clustering/clustering_tools.py", line 655, in detect_mixtures
    del recording, sub_recording, local_params, templates
                   ^^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'sub_recording' where it is not associated with a value

Spike sorting failed. You can inspect the runtime trace in /media/paul/storage/Data/test/processed/results_sc/6/spikeinterface_log.json.

[zm711]

Hey Paul, @yger can comment on this. But first could you let us know the version of spikeinterface. I've recently run into a similar issue with run_sorter_by_property so I want to try to also track the version at which this might be happening.

[paul-aparicio]

Thanks! I'm running spikeinterface version 0.102.2

[zm711]

Could you try the exact same code with 0.102.1 to see if that fixes it? I think we may have a bug between those versions. I'm trying to track it down too! See #3814 where I was able to create a sorting error in MS5.

[paul-aparicio]

Got a very similar error with version 0.102.1:

Traceback (most recent call last):
  File "/home/paul/anaconda3/envs/sin/lib/python3.11/site-packages/spikeinterface/sorters/basesorter.py", line 270, in run_from_folder
    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)
  File "/home/paul/anaconda3/envs/sin/lib/python3.11/site-packages/spikeinterface/sorters/internal/spyking_circus2.py", line 278, in _run_from_folder
    labels, peak_labels = find_cluster_from_peaks(
                          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paul/anaconda3/envs/sin/lib/python3.11/site-packages/spikeinterface/sortingcomponents/clustering/main.py", line 44, in find_cluster_from_peaks
    outputs = method_class.main_function(recording, peaks, params, job_kwargs=job_kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paul/anaconda3/envs/sin/lib/python3.11/site-packages/spikeinterface/sortingcomponents/clustering/circus.py", line 310, in main_function
    labels, peak_labels = remove_duplicates_via_matching(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paul/anaconda3/envs/sin/lib/python3.11/site-packages/spikeinterface/sortingcomponents/clustering/clustering_tools.py", line 666, in remove_duplicates_via_matching
    similar_templates = detect_mixtures(
                        ^^^^^^^^^^^^^^^^
  File "/home/paul/anaconda3/envs/sin/lib/python3.11/site-packages/spikeinterface/sortingcomponents/clustering/clustering_tools.py", line 655, in detect_mixtures
    del recording, sub_recording, local_params, templates
                   ^^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'sub_recording' where it is not associated with a value

[zm711]

Hey @paul-aparicio so this turned out to be unrelated to my error.

https://github.com/SpikeInterface/spikeinterface/blob/a95dc57d1a8c71e6c751f5d5e8b4404dd7573eb5/src/spikeinterface/sortingcomponents/clustering/clustering_tools.py#L597

if we look here we see that we try to run a sub_recording only if you have some number of templates nearby. In your case we must have a well isolated template so we are trying to delete a variable that is never created to reduce memory footprint. I'll submit a draft PR and then @yger and @samuelgarcia can review it :)

[zm711]

Are you willing to check out the linked PR #3820?

[paul-aparicio]

Sounds good, I will give it a try!

[paul-aparicio]

I get a similar error after the PR (pasted below).

Traceback (most recent call last):
  File "/home/paul/anaconda3/envs/sind/lib/python3.11/site-packages/spikeinterface/sorters/basesorter.py", line 270, in run_from_folder
    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)
  File "/home/paul/anaconda3/envs/sind/lib/python3.11/site-packages/spikeinterface/sorters/internal/spyking_circus2.py", line 278, in _run_from_folder
    labels, peak_labels = find_cluster_from_peaks(
                          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paul/anaconda3/envs/sind/lib/python3.11/site-packages/spikeinterface/sortingcomponents/clustering/main.py", line 44, in find_cluster_from_peaks
    outputs = method_class.main_function(recording, peaks, params, job_kwargs=job_kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paul/anaconda3/envs/sind/lib/python3.11/site-packages/spikeinterface/sortingcomponents/clustering/circus.py", line 310, in main_function
    labels, peak_labels = remove_duplicates_via_matching(
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paul/anaconda3/envs/sind/lib/python3.11/site-packages/spikeinterface/sortingcomponents/clustering/clustering_tools.py", line 666, in remove_duplicates_via_matching
    similar_templates = detect_mixtures(
                        ^^^^^^^^^^^^^^^^
  File "/home/paul/anaconda3/envs/sind/lib/python3.11/site-packages/spikeinterface/sortingcomponents/clustering/clustering_tools.py", line 655, in detect_mixtures
    del recording, sub_recording, local_params, templates
                   ^^^^^^^^^^^^^
UnboundLocalError: cannot access local variable 'sub_recording' where it is not associated with a value ```

[zm711]

That's interesting because sub_recording has to be defined in that PR. How did you install from the PR? I just want to make sure you got the actual changes incorporated :)

[paul-aparicio]

Thanks for checking! I am installing spikeinterface with anaconda. So I deleted the environment, installed all the dependencies basically using the yml file from the "installation tips" section, but excluding spikeinterface. Then I clone the repository and installed spikeinterface from the code. Then I use git status to check that code is ok. Then I use gh pr checkout 3820

You comment made me realize that I still needed to reinstall the updated code, so I uninstalled spikeinterface using pip, and then reinstalled. The version it gives me 0.102.2, which I thought we needed 0.102.1? But rerunning my code again produced a different error, so maybe I finally figured out how to install it correctly!

SpikeSortingError: Spike sorting error trace:
Traceback (most recent call last):
  File "/home/paul/anaconda3/envs/sind/lib/python3.11/site-packages/spikeinterface-0.102.2-py3.11.egg/spikeinterface/sorters/basesorter.py", line 270, in run_from_folder
    SorterClass._run_from_folder(sorter_output_folder, sorter_params, verbose)
  File "/home/paul/anaconda3/envs/sind/lib/python3.11/site-packages/spikeinterface-0.102.2-py3.11.egg/spikeinterface/sorters/internal/spyking_circus2.py", line 293, in _run_from_folder
    templates_array = estimate_templates(
                      ^^^^^^^^^^^^^^^^^^^
  File "/home/paul/anaconda3/envs/sind/lib/python3.11/site-packages/spikeinterface-0.102.2-py3.11.egg/spikeinterface/core/waveform_tools.py", line 754, in estimate_templates
    templates_array = estimate_templates_with_accumulator(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/paul/anaconda3/envs/sind/lib/python3.11/site-packages/spikeinterface-0.102.2-py3.11.egg/spikeinterface/core/waveform_tools.py", line 828, in estimate_templates_with_accumulator
    assert spikes.size > 0, "estimate_templates() need non empty sorting"
           ^^^^^^^^^^^^^^^
AssertionError: estimate_templates() need non empty sorting

[zm711]

Yes you installed it correctly. The dev_mode for this will be 0.102.2 (0.102.1 is "frozen" as a release so we don't modify that any more :) )

Yeah so I think this at least got you to a useful error. It is saying that the sorting that was initially generated for this step was empty so that means either 1) the input data could have a problem or 2) spykingcircus 2 would need to work on handling this type of data. Have you tried another sorter to see if another sorter worked with this data?

[paul-aparicio]

Yes, I get ironclust, mountainsort4/5 and waveclus to work through the data (shooting to get four to work to try and take advantage of the comparison sorting described in the paper). This data has lots of brief drop out (its from a wireless headstage).

[zm711]

I'm reopening this because I want @yger to take a quick look. To summarize for Pierre, other spike sorters work on this data but it appears SC2 is failing. If you want to address this and help @paul-aparicio that would be great. Otherwise @paul-aparicio you could try tridesclous2 or KS4 if you want to try to get to a fourth sorter!

[yger]

Yes, I can have a look, but could I get the data? We are actively working with Sam on the components, so SC2 is excpected to change a bit in the next months, especially concerning its clustering. THe problem here seems that no spikes are detected, and thus the code is crashing. But this is strange, so I need to troubleshoot

[paul-aparicio]

sure, The blackrock file is ~20GB. How would you like me to get it to you? Sadly, Tridesclous and KS4 typically don't work on our data (probably for the same reason). Spyking Circus usually works for me though. Thanks for the help!

[yger]

Up to you, do you have a provider to upload such large file? Otherwise I can send you a link

[paul-aparicio]

A link would probably be easiest. Thanks!

[yger]

Can you send me your email at [email protected] such that I can send it?