Error in Singularity container: numcodecs.blosc import failure

[JerryChen1974]

Description:

I am encountering an issue when running spike sorting on large datasets (278GB) with Kilosort4 in a Singularity container. The same pipeline works perfectly with smaller datasets (2GB), but fails with this larger one.

Environment:

• Platform: MIT OpenMind computing cluster
• Python version: 3.11
• SpikeInterface version: 0.102.1
• Singularity version: 3.9.5

Issue:

The sorting step fails with the following error:

/net/vast-storage/scratch/vast/feelab/jchen13/openmind-sorting/sorting.py:55: DeprecationWarning: load_extractor() is deprecated and will be removed in the future. Please use load() instead.
preprocessed_recordings[group] = si.load_extractor(recording_path)
Traceback (most recent call last):
File "/net/vast-storage/scratch/vast/feelab/jchen13/openmind-sorting/sorting.py", line 120, in <module>
main(args)
File "/net/vast-storage/scratch/vast/feelab/jchen13/openmind-sorting/sorting.py", line 102, in main
sorting = si.run_sorter(
^^^^^^^^^^^^^^
File "/home/jchen13/.conda/envs/si_new/lib/python3.11/site-packages/spikeinterface/sorters/runsorter.py", line 193, in run_sorter
return run_sorter_container(
^^^^^^^^^^^^^^^^^^^^^
File "/home/jchen13/.conda/envs/si_new/lib/python3.11/site-packages/spikeinterface/sorters/runsorter.py", line 648, in run_sorter_container
raise SpikeSortingError(f"Spike sorting in {mode} failed with the following error:\n{run_sorter_output}")
spikeinterface.sorters.utils.misc.SpikeSortingError: Spike sorting in singularity failed with the following error:
Traceback (most recent call last):
File "/net/vast-storage/scratch/vast/feelab/jchen13/sorting_outputs/outputs_11208/2025-02-19_11208_Post-Advance_Nidopallium_BOTM_0/ks_outputs/in_container_sorter_script.py", line 4, in <module>
from spikeinterface import load
File "/net/vast-storage/scratch/vast/feelab/jchen13/sorting_outputs/outputs_11208/2025-02-19_11208_Post-Advance_Nidopallium_BOTM_0/ks_outputs/in_container_python_base/lib/python3.11/site-packages/spikeinterface/**init**.py", line 7, in <module>
from .core import *
File "/net/vast-storage/scratch/vast/feelab/jchen13/sorting_outputs/outputs_11208/2025-02-19_11208_Post-Advance_Nidopallium_BOTM_0/ks_outputs/in_container_python_base/lib/python3.11/site-packages/spikeinterface/core/**init**.py", line 20, in <module>
from .zarrextractors import ZarrRecordingExtractor, ZarrSortingExtractor, read_zarr, get_default_zarr_compressor
File "/net/vast-storage/scratch/vast/feelab/jchen13/sorting_outputs/outputs_11208/2025-02-19_11208_Post-Advance_Nidopallium_BOTM_0/ks_outputs/in_container_python_base/lib/python3.11/site-packages/spikeinterface/core/zarrextractors.py", line 6, in <module>
import zarr
File "/net/vast-storage/scratch/vast/feelab/jchen13/sorting_outputs/outputs_11208/2025-02-19_11208_Post-Advance_Nidopallium_BOTM_0/ks_outputs/in_container_python_base/lib/python3.11/site-packages/zarr/**init**.py", line 3, in <module>
from zarr.convenience import (
File "/net/vast-storage/scratch/vast/feelab/jchen13/sorting_outputs/outputs_11208/2025-02-19_11208_Post-Advance_Nidopallium_BOTM_0/ks_outputs/in_container_python_base/lib/python3.11/site-packages/zarr/convenience.py", line 8, in <module>
from zarr._storage.store import data_root, meta_root, assert_zarr_v3_api_available
File "/net/vast-storage/scratch/vast/feelab/jchen13/sorting_outputs/outputs_11208/2025-02-19_11208_Post-Advance_Nidopallium_BOTM_0/ks_outputs/in_container_python_base/lib/python3.11/site-packages/zarr/_storage/store.py", line 10, in <module>
from zarr.meta import Metadata2, Metadata3
File "/net/vast-storage/scratch/vast/feelab/jchen13/sorting_outputs/outputs_11208/2025-02-19_11208_Post-Advance_Nidopallium_BOTM_0/ks_outputs/in_container_python_base/lib/python3.11/site-packages/zarr/meta.py", line 10, in <module>
from zarr.util import json_dumps, json_loads
File "/net/vast-storage/scratch/vast/feelab/jchen13/sorting_outputs/outputs_11208/2025-02-19_11208_Post-Advance_Nidopallium_BOTM_0/ks_outputs/in_container_python_base/lib/python3.11/site-packages/zarr/util.py", line 34, in <module>
from numcodecs.blosc import cbuffer_sizes, cbuffer_metainfo
ImportError: cannot import name 'cbuffer_sizes' from 'numcodecs.blosc' (/net/vast-storage/scratch/vast/feelab/jchen13/sorting_outputs/outputs_11208/2025-02-19_11208_Post-Advance_Nidopallium_BOTM_0/ks_outputs/in_container_python_base/sclib/python3.11/site-packages/numcodecs/blosc.cpython-311-x86_64-linux-gnu.so)

This is how I am calling the sorter with Singularity in my sorting.py file which is attached to this issue:

 # Run the sorter
        sorting = si.run_sorter(
            sorter_name='spikeinterface/kilosort4-base:4.0.18_cuda-12.0.0',
            recording=sub_recording,
            folder=ks_sub_folder,
            singularity_image=True, # Update as necessary
            verbose=True,
            remove_existing_folder=True,
            **params
            )

sorting.py.pdf

Any guidance on how to resolve this issue would be greatly appreciated. We're happy to provide additional information if needed.

[alejoe91]

Hi @JerryChen1974

I don't think this is due to size, but rather to a recent breaking change of numcodecs (see https://github.com/SpikeInterface/spikeinterface/pull/3845)

Make sure your python environment is not using the latest numcodecs version, since this is causing issues.