spikeinterface
spikeinterface copied to clipboard
SpikeInterface
Error running SpykingCircus2 clustering
Hi @yger,
I have a question about the clustering process in spyking circus. I noticed that sometimes when I run the sorter on small periods of data or channels with a few peaks I get the following error. I assumed that it was coming from a lack of spikes to perform a clustering but I would like to make sure it is not coming from something else.
Thank you, Anthony
If this is a small recording, can you share it with me (alongside with the analysis script)?
Of course @yger , here is a link to download the dataset : https://instituteicm-my.sharepoint.com/:f:/g/personal/anthony_pinto_icm-institute_org/EpiPint_OABJiQNFPG2Q-44B7cmBfe_iV1CJOI_RrHLZKg?e=XCZgWL
It is already filtered using a bandpass filter between 300 and 6000Hz with dtype64 so I did not put apply preprocessing in the run_sorter function.
The sorting is integrated in a pipeline so it is difficult to give you the full code but I made the following code that is replicating the process to reproduce the error. Here is the code :
from spikeinterface.sorters import run_sorter
import spikeinterface as si
recording = si.load_extractor(r'C:\Users\anthony.pinto\Documents\dataset_fail_clustering\recording.zarr')
params = si.sorters.get_default_sorter_params('spykingcircus2')
params['waveforms']['max_spikes_per_unit'] = 500
params['detection']['peak_sign'] = 'pos'
params['detection']['detect_threshold'] = 3.5779258099871694
params['apply_preprocessing'] = False
sorting = run_sorter('spykingcircus2', recording, output_folder='test_run_sorting', **params)
The weird value of the detection threshold is determined automatically so don't mind it. I didn't tweak the other parameters appart from those in the above code, maybe I should but I'm not sure of what each parameter is doing so that is why I didn't change a lot of the params.
Thank you, Anthony
So on my machine, with the main branch, the code is working fine. This is both a good and a bad news :-( Can you tell me the version of hdbscan that you have?
Argh, yes ofc here is my hdbscan version :
Note also that I get this working with both Windows and Linux.
When you say "you get this working", you mean that the code is crashing? I have the sam hdbscan version, so maybe send me the full version for the envs (pip list)
Yes sorry for the formulation, I mean the code crashes.
Here is my pip list :
Package Version Editable project location
------------------------- ------------ ------------------------------------------
aiohttp 3.9.1
aiohttp-retry 2.8.3
aiosignal 1.3.1
altair 5.0.1
amqp 5.2.0
annotated-types 0.6.0
antlr4-python3-runtime 4.9.3
antropy 0.1.5
anyio 4.1.0
appdirs 1.4.4
argon2-cffi 23.1.0
argon2-cffi-bindings 21.2.0
arrow 1.3.0
asciitree 0.3.3
asttokens 2.4.1
async-timeout 4.0.3
asyncssh 2.14.1
atomicwrites 1.4.1
atpublic 4.0
attrs 23.1.0
audioread 3.0.1
autoreject 0.4.3
backcall 0.2.0
beautifulsoup4 4.12.2
billiard 4.2.0
black 23.11.0
bleach 6.1.0
cachetools 5.3.2
cbor2 5.4.6
celery 5.3.6
certifi 2023.11.17
cffi 1.16.0
cfgv 3.4.0
chardet 5.2.0
charset-normalizer 3.3.2
click 8.1.7
click-didyoumean 0.3.0
click-plugins 1.1.1
click-repl 0.3.0
cloudpickle 3.0.0
colorama 0.4.6
colorcet 3.0.1
coloredlogs 15.0.1
comm 0.2.0
configobj 5.0.8
contourpy 1.2.0
coverage 7.3.2
cryptography 41.0.7
cycler 0.12.1
Cython 0.29.36
dacite 1.8.1
dask 2023.11.0
debugpy 1.8.0
decorator 5.1.1
defusedxml 0.7.1
dictdiffer 0.9.0
diskcache 5.6.3
distlib 0.3.7
distributed 2023.5.1
distro 1.8.0
dpath 2.1.6
dulwich 0.21.6
dvc 3.33.4
dvc-data 2.24.0
dvc-http 2.30.2
dvc-objects 2.0.1
dvc-render 1.0.0
dvc-studio-client 0.18.0
dvc-task 0.3.0
dvclive 3.5.1
EDFlib-Python 1.0.7
elephant 0.12.0
entrypoints 0.4
exceptiongroup 1.2.0
executing 2.0.1
fasteners 0.19
fastjsonschema 2.19.0
figurl 0.2.18
filelock 3.13.1
flake8 4.0.1
flakeheaven 3.3.0
flatbuffers 23.5.26
flatten-dict 0.4.2
flufl.lock 7.1.1
fonttools 4.45.1
fqdn 1.5.1
frozenlist 1.4.0
fsspec 2023.10.0
funcy 2.0
ghp-import 2.1.0
gitdb 4.0.11
GitPython 3.1.40
grandalf 0.8
gto 1.4.0
h5io 0.1.9
h5py 3.10.0
hdbscan 0.8.33
hdmf 3.6.1
htmlmin 0.1.12
humanfriendly 10.0
hydra-core 1.3.2
identify 2.5.32
idna 3.6
ImageHash 4.3.1
importlib-metadata 6.8.0
importlib-resources 6.1.1
iniconfig 2.0.0
ipykernel 6.27.1
ipython 8.18.1
ipython-genutils 0.2.0
ipywidgets 8.1.1
isoduration 20.11.0
iterative-telemetry 0.0.8
jedi 0.19.1
Jinja2 3.1.2
joblib 1.3.2
jsonpointer 2.4
jsonschema 4.20.0
jsonschema-specifications 2023.11.1
jupyter_client 7.4.9
jupyter_core 5.5.0
jupyter-events 0.9.0
jupyter_server 2.11.1
jupyter_server_terminals 0.4.4
jupyterlab_pygments 0.3.0
jupyterlab-widgets 3.0.9
kachery-cloud 0.4.2
kiwisolver 1.4.5
kombu 5.3.4
lazy_loader 0.3
librosa 0.10.1
lightgbm 4.1.0
littleutils 0.2.2
llvmlite 0.39.1
locket 1.0.0
lspopt 1.3.0
Mako 1.3.0
Markdown 3.5.1
markdown-it-py 3.0.0
MarkupSafe 2.1.3
mat73 0.60
matplotlib 3.8.2
matplotlib-inline 0.1.6
mccabe 0.6.1
mdurl 0.1.2
mergedeep 1.3.4
mistune 3.0.2
mkdocs 1.5.3
mne 1.6.0
mne-features 0.2.1
mpmath 1.3.0
msgpack 1.0.7
mtscomp 1.0.2
multidict 6.0.4
multimethod 1.10
mypy 1.7.1
mypy-extensions 1.0.0
nanotime 0.5.2
nbclassic 1.0.0
nbclient 0.9.0
nbconvert 7.11.0
nbformat 5.9.2
neo 0.12.0
nest-asyncio 1.5.8
networkx 3.2.1
nodeenv 1.8.0
notebook 6.5.6
notebook_shim 0.2.3
numba 0.56.4
numcodecs 0.12.1
numexpr 2.8.7
numpy 1.23.5
omegaconf 2.3.0
onnx 1.15.0
onnxconverter-common 1.13.0
onnxmltools 1.11.2
onnxruntime 1.16.1
orjson 3.9.10
outdated 0.2.2
overrides 7.4.0
packaging 23.2
pandas 2.1.3
pandocfilters 1.5.0
param 2.0.1
parso 0.8.3
partd 1.4.1
pathspec 0.11.2
patsy 0.5.3
pdoc3 0.10.0
phik 0.12.3
phy 2.0b5
phylib 2.4.3
pickleshare 0.7.5
Pillow 9.5.0
pip 23.1.2
platformdirs 3.11.0
pluggy 1.3.0
pooch 1.8.0
pre-commit 3.5.0
priam 0.1.0 C:\Users\anthony.pinto\Documents\git\priam
probeinterface 0.2.19
prometheus-client 0.19.0
prompt-toolkit 3.0.41
protobuf 4.25.1
psutil 5.9.6
pubnub 6.3.3
pure-eval 0.2.2
py 1.11.0
pyarrow 11.0.0
pycodestyle 2.8.0
pycparser 2.21
pycryptodomex 3.18.0
pyct 0.5.0
pydantic 2.5.2
pydantic_core 2.14.5
pydot 1.4.2
pyflakes 2.4.0
pygit2 1.13.3
Pygments 2.17.2
pygtrie 2.5.0
pymatreader 0.0.32
pynwb 2.3.2
PyOpenGL 3.1.1a1
PyOpenGL-accelerate 3.1.7
pyparsing 3.1.1
pyprep 0.4.3
pyproject-api 1.6.1
PyQt5 5.15.10
PyQt5-Qt5 5.15.2
PyQt5-sip 12.13.0
PyQtWebEngine 5.15.6
PyQtWebEngine-Qt5 5.15.2
pyreadline3 3.4.1
pyriemann 0.5
pyrsistent 0.19.3
pytest 7.4.3
pytest-cov 4.1.0
pytest-flakes 4.0.5
pytest-html 3.2.0
pytest-metadata 3.0.0
python-dateutil 2.8.2
python-json-logger 2.0.7
python-picard 0.7
pytz 2023.3.post1
PyWavelets 1.5.0
pywin32 306
pywinpty 2.0.12
PyYAML 6.0.1
pyyaml_env_tag 0.1
pyzmq 24.0.1
qtconsole 5.5.1
QtPy 2.4.1
quantities 0.14.1
referencing 0.31.0
requests 2.31.0
rfc3339-validator 0.1.4
rfc3986-validator 0.1.1
rich 13.7.0
rpds-py 0.13.1
ruamel.yaml 0.18.5
ruamel.yaml.clib 0.2.8
scikit-learn 1.2.2
scipy 1.11.4
scmrepo 1.6.0
seaborn 0.13.0
semver 3.0.2
Send2Trash 1.8.2
setuptools 69.0.2
setuptools-scm 8.0.4
shortuuid 1.0.11
shtab 1.6.4
simplejson 3.19.1
six 1.16.0
skl2onnx 1.15.0
sleepecg 0.5.5
smmap 5.0.1
sniffio 1.3.0
sortedcontainers 2.4.0
sortingview 0.11.15
soundfile 0.12.1
soupsieve 2.5
soxr 0.3.7
spikeinterface 0.100.0.dev0
sqltrie 0.9.0
stack-data 0.6.3
statsmodels 0.14.0
stochastic 0.6.0
sympy 1.12
tabulate 0.9.0
tangled-up-in-unicode 0.2.0
tblib 1.7.0
tensorpac 0.6.5
terminado 0.18.0
threadpoolctl 3.2.0
tinycss2 1.2.1
tk 0.1.0
toml 0.10.2
tomli 2.0.1
tomlkit 0.12.3
toolz 0.12.0
tornado 6.3.3
tox 4.11.4
tqdm 4.66.1
traitlets 5.14.0
typeguard 2.13.3
typer 0.9.0
types-python-dateutil 2.8.19.14
typing_extensions 4.8.0
tzdata 2023.3
uri-template 1.3.0
urllib3 2.1.0
vine 5.1.0
virtualenv 20.24.7
visions 0.7.5
viziphant 0.3.0
voluptuous 0.14.1
watchdog 3.0.0
wcwidth 0.2.12
webcolors 1.13
webencodings 0.5.1
websocket-client 1.6.4
wheel 0.40.0
widgetsnbextension 4.0.9
wordcloud 1.9.2
xmltodict 0.13.0
yarl 1.9.3
yasa 0.6.3
ydata-profiling 4.5.1
zarr 2.16.1
zc.lockfile 3.0.post1
zict 3.0.0
zipp 3.17.0
Hi Anthony,
I encountered a similar problem with my single-channel data today.
I had installed SpikeInterface several months ago, and in the version I had 0.99.1 directly installed via pip, the following lines of code in sortingcomponents/clustering/random_projections.py
projections = np.random.randn(num_chans, d["nb_projections"])
projections -= projections.mean(0)
projections /= projections.std(0)
resulted in all NaN values in the 'projections' array, which then caused a ValueError. Then I found that this issue has been fixed in the current version of the SpikeInterface repository, as seen in this commit: https://github.com/SpikeInterface/spikeinterface/commit/93fbc55a29d769020b4835156dd1bfe094e8296c
I noticed that you also dealing with single-channel data, and I'm curious if you experienced a similar problem due to the same reason. If so, the main branch version works fine!
Hi @wliWanru,
Thanks for this update, I will try to update spikeinterface and see if this works. Thank you
Since this is over one year old and there have been updates to SI and SC2 I'll close this issue and feel free to open a new issue if you have other problems with SI or SC2.
