kilosort3 on dandihub

[bendichter]

I’m having trouble running kilosort3 singularity on dandi hub.

!pip install git+https://github.com/SpikeInterface/spikeinterface


import spikeinterface.sorters as ss
from spikeinterface.core.testing_tools import generate_recording

recording = generate_recording(
    num_channels=32,
    sampling_frequency=30000.,  # in Hz
    durations=[120],
    set_probe=True,
    ndim=2,
)

recording2 = recording.save(folder="test_recording2")

write_binary_recording with n_jobs 1  chunk_size None
---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Input In [9], in <module>
      4 recording = generate_recording(
      5     num_channels=32,
      6     sampling_frequency=30000.,  # in Hz
   (...)
      9     ndim=2,
     10 )
     12 recording2 = recording.save(folder="test_recording2")
---> 13 sorting = ss.run_kilosort3(recording2, singularity_image=True, verbose=True)

File /opt/conda/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py:611, in run_kilosort3(*args, **kwargs)
    610 def run_kilosort3(*args, **kwargs):
--> 611     return run_sorter('kilosort3', *args, **kwargs)

File /opt/conda/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py:108, in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, with_output, **sorter_params)
     95 common_kwargs = dict(
     96     sorter_name=sorter_name,
     97     recording=recording,
   (...)
    104     **sorter_params,
    105 )
    107 if docker_image or singularity_image:
--> 108     return run_sorter_container(
    109         container_image=docker_image if isinstance(docker_image, str) else singularity_image,
    110         mode="docker" if docker_image else "singularity",
    111         **common_kwargs,
    112     )
    114 return run_sorter_local(**common_kwargs)

File /opt/conda/lib/python3.9/site-packages/spikeinterface/sorters/runsorter.py:397, in run_sorter_container(sorter_name, recording, mode, container_image, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, extra_requirements, **sorter_params)
    395 if use_gpu:
    396     if gpu_capability == 'nvidia-required':
--> 397         assert has_nvidia(), "The container requires a NVIDIA GPU capability, but it is not available"
    398         extra_kwargs['container_requires_gpu'] = True
    399     elif gpu_capability == 'nvidia-optional':

AssertionError: The container requires a NVIDIA GPU capability, but it is not available

I did start a GPU instance. I tried again just to make sure. Any ideas what might be going on here? @chyumin , you were able to git ks3 to run on dandi hub, right?

[alejoe91]

@bendichter just to double check you get normal output for nvidia-smi?

[chyumin]

@bendichter PR #674 inserts a test to check if there is a nvidia gpu installed by trying to run nvidia-smi bash command (here). I recall that in dandihub we can't run this command. Needs to be double checked

[alejoe91]

Ahhhh that might explain it!

[chyumin]

We need to find another way to test nvidia gpu availability

[alejoe91]

Any update on this?

[h-mayorquin]

I think we can use the following function:

from cuda import cuda

def has_cuda_capable_device():

  try:
    cu_result_init,  = cuda.cuInit(0)
    cu_result, cu_string = cuda.cuGetErrorString(cu_result_init)
    cu_result_device_count, device_count = cuda.cuDeviceGetCount()
    return device_count > 0
  except RuntimeError: #  Failed to dlopen libcuda.so
    return False
    

See https://github.com/dandi/dandi-hub/issues/36 for more context

[alejoe91]

fixed by #1398