Problem of Spike Sorting in Concatenated Recordings

[farazmoradi]

Hi,

I receive this error while spike sorting with Kilosort on concatenated recordings:

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Cell In[5], line 2
      1 #multirecording = spre.common_reference(multirecording,reference='global')
----> 2 sorting_TDC_5 = ss.run_sorter("kilosort2", recording=multirecording, output_folder=r"G:\T9 test")
      3 sorting_TDC_5.get_unit_ids()

File [c:\Users\BeiqueLab\anaconda3\Lib\site-packages\spikeinterface\sorters\runsorter.py:148](file:///C:/Users/BeiqueLab/anaconda3/Lib/site-packages/spikeinterface/sorters/runsorter.py:148), in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, delete_container_files, with_output, **sorter_params)
    141             container_image = singularity_image
    142     return run_sorter_container(
    143         container_image=container_image,
    144         mode=mode,
    145         **common_kwargs,
    146     )
--> 148 return run_sorter_local(**common_kwargs)

File [c:\Users\BeiqueLab\anaconda3\Lib\site-packages\spikeinterface\sorters\runsorter.py:173](file:///C:/Users/BeiqueLab/anaconda3/Lib/site-packages/spikeinterface/sorters/runsorter.py:173), in run_sorter_local(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, **sorter_params)
    171 output_folder = SorterClass.initialize_folder(recording, output_folder, verbose, remove_existing_folder)
    172 SorterClass.set_params_to_folder(recording, output_folder, sorter_params, verbose)
--> 173 SorterClass.setup_recording(recording, output_folder, verbose=verbose)
    174 SorterClass.run_from_folder(output_folder, raise_error, verbose)
    175 if with_output:

File [c:\Users\BeiqueLab\anaconda3\Lib\site-packages\spikeinterface\sorters\basesorter.py:206](file:///C:/Users/BeiqueLab/anaconda3/Lib/site-packages/spikeinterface/sorters/basesorter.py:206), in BaseSorter.setup_recording(cls, recording, output_folder, verbose)
    204     all_params = json.load(f)
...
--> 303 file.truncate(file_size_bytes)
    304 file.close()
    305 assert Path(file_path).is_file()

OSError: [Errno 22] Invalid argument
Output is truncated. 

[alejoe91]

Hi @farazmoradi

Can you paster the full error?

[farazmoradi]

Hello @alejoe91

I pasted the full error that I receive in VS code.

[farazmoradi]

c:\Users\BeiqueLab\anaconda3\Lib\site-packages\spikeinterface\core\core_tools.py:299: RuntimeWarning: overflow encountered in scalar multiply
  data_size_bytes = dtype_size_bytes * num_frames * num_channels
---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Cell In[4], line 2
      1 #multirecording = spre.common_reference(multirecording,reference='global')
----> 2 sorting_TDC_5 = ss.run_sorter("kilosort2", recording=multirecording, output_folder=r"G:\T9 test")
      3 sorting_TDC_5.get_unit_ids()

File c:\Users\BeiqueLab\anaconda3\Lib\site-packages\spikeinterface\sorters\runsorter.py:148, in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, delete_container_files, with_output, **sorter_params)
    141             container_image = singularity_image
    142     return run_sorter_container(
    143         container_image=container_image,
    144         mode=mode,
    145         **common_kwargs,
    146     )
--> 148 return run_sorter_local(**common_kwargs)

File c:\Users\BeiqueLab\anaconda3\Lib\site-packages\spikeinterface\sorters\runsorter.py:173, in run_sorter_local(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, **sorter_params)
    171 output_folder = SorterClass.initialize_folder(recording, output_folder, verbose, remove_existing_folder)
    172 SorterClass.set_params_to_folder(recording, output_folder, sorter_params, verbose)
--> 173 SorterClass.setup_recording(recording, output_folder, verbose=verbose)
    174 SorterClass.run_from_folder(output_folder, raise_error, verbose)
    175 if with_output:

File c:\Users\BeiqueLab\anaconda3\Lib\site-packages\spikeinterface\sorters\basesorter.py:206, in BaseSorter.setup_recording(cls, recording, output_folder, verbose)
    204     all_params = json.load(f)
...
--> 303 file.truncate(file_size_bytes)
    304 file.close()
    305 assert Path(file_path).is_file()

OSError: [Errno 22] Invalid argument
Output is truncated. View as a scrollable element or open in a text editor. Adjust cell output settings...

[alejoe91]

Vs code truncates the printed output (see the ...). Can you run it in s Jupyter notebook?

[farazmoradi]

Thanks for the info!

---------------------------------------------------------------------------
OSError                                   Traceback (most recent call last)
Cell In[4], line 2
      1 #multirecording = spre.common_reference(multirecording,reference='global')
----> 2 sorting_TDC_5 = ss.run_sorter("kilosort2", recording=multirecording, output_folder=r"G:\T9 test")
      3 sorting_TDC_5.get_unit_ids()

File ~\anaconda3\Lib\site-packages\spikeinterface\sorters\runsorter.py:148, in run_sorter(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, docker_image, singularity_image, delete_container_files, with_output, **sorter_params)
    141             container_image = singularity_image
    142     return run_sorter_container(
    143         container_image=container_image,
    144         mode=mode,
    145         **common_kwargs,
    146     )
--> 148 return run_sorter_local(**common_kwargs)

File ~\anaconda3\Lib\site-packages\spikeinterface\sorters\runsorter.py:173, in run_sorter_local(sorter_name, recording, output_folder, remove_existing_folder, delete_output_folder, verbose, raise_error, with_output, **sorter_params)
    171 output_folder = SorterClass.initialize_folder(recording, output_folder, verbose, remove_existing_folder)
    172 SorterClass.set_params_to_folder(recording, output_folder, sorter_params, verbose)
--> 173 SorterClass.setup_recording(recording, output_folder, verbose=verbose)
    174 SorterClass.run_from_folder(output_folder, raise_error, verbose)
    175 if with_output:

File ~\anaconda3\Lib\site-packages\spikeinterface\sorters\basesorter.py:206, in BaseSorter.setup_recording(cls, recording, output_folder, verbose)
    204     all_params = json.load(f)
    205     sorter_params = all_params["sorter_params"]
--> 206 cls._setup_recording(recording, sorter_output_folder, sorter_params, verbose)

File ~\anaconda3\Lib\site-packages\spikeinterface\sorters\external\kilosortbase.py:153, in KilosortBase._setup_recording(cls, recording, sorter_output_folder, params, verbose)
    151     else:
    152         padded_recording = recording
--> 153     write_binary_recording(
    154         recording=padded_recording,
    155         file_paths=[binary_file_path],
    156         dtype="int16",
    157         **get_job_kwargs(params, verbose),
    158     )
    160 cls._generate_ops_file(recording, params, sorter_output_folder, binary_file_path)

File ~\anaconda3\Lib\site-packages\spikeinterface\core\core_tools.py:303, in write_binary_recording(recording, file_paths, dtype, add_file_extension, byte_offset, auto_cast_uint, **job_kwargs)
    300 file_size_bytes = data_size_bytes + byte_offset
    302 file = open(file_path, "wb+")
--> 303 file.truncate(file_size_bytes)
    304 file.close()
    305 assert Path(file_path).is_file()

OSError: [Errno 22] Invalid argument

[farazmoradi]

Hello @alejoe91, is there any news?

[alejoe91]

@farazmoradi unfortunately not. @h-mayorquin can you have a look when you have some time?

[h-mayorquin]

Hi @farazmoradi . Sorry for not coming to you on this. It must habe slipped through the cracks.

That function has changed since the last time we ran it. Can I ask you kindly to try again with the latest version of the repo?

Could you share the data with us? That will be the easier way for us to debug this. If that is not possible, could you show what is the value of fil_size_bytes in the code that produces the error:

    300 file_size_bytes = data_size_bytes + byte_offset
    302 file = open(file_path, "wb+")
--> 303 file.truncate(file_size_bytes)
    304 file.close()

I suspect that one of these values (that are a little bit before in the code there) is wrong:

        num_frames = recording.get_num_frames(segment_index=segment_index)
        data_size_bytes = dtype_size_bytes * num_frames * num_channels
        file_size_bytes = data_size_bytes + byte_offset

So it will be good to see them.

[JoeZiminski]

Hey @farazmoradi, did you have any luck with the new version on the repo?

[h-mayorquin]

I think we should close this by inactivity if the OP does not reply on a week.