Multiprocessing of si_based_sorters fails with TypeError: cannot pickle 'weakref' object

[hornauerp]

trafficstars

Hello,

Running Spyking Circus 2 (same goes for Tridesclous 2) on both a real dataset and a recording generated by si.toy_example() fail with a pickle error. Here the minimal reproducible example using a toy example:

import spikeinterface.full as si
recording, sorting_true = si.toy_example(duration=10, seed=0, num_segments=1)
recording = recording.save()
sorting_well_1 = si.run_sorter('tridesclous2', recording)`

This results in the following error message:

Use cache_folder=C:\Users\Philipp\AppData\Local\Temp\spikeinterface_cache\tmpaj5nlsdk\DPUSRRVC
write_binary_recording with n_jobs = 1 and chunk_size = None
Traceback (most recent call last):
  File "E:\envs\si\lib\site-packages\spikeinterface-0.94.1.dev0-py3.10.egg\spikeinterface\sorters\basesorter.py", line 225, in run_from_folder
    SorterClass._run_from_folder(output_folder, sorter_params, verbose)
  File "E:\envs\si\lib\site-packages\spikeinterface-0.94.1.dev0-py3.10.egg\spikeinterface\sorters\si_based_sorters\tridesclous2.py", line 68, in _run_from_folder
    peaks = detect_peaks(recording, method='locally_exclusive',  **detection_params, **job_kwargs)
  File "E:\envs\si\lib\site-packages\spikeinterface-0.94.1.dev0-py3.10.egg\spikeinterface\sortingcomponents\peak_detection.py", line 117, in detect_peaks
    outputs = processor.run()
  File "E:\envs\si\lib\site-packages\spikeinterface-0.94.1.dev0-py3.10.egg\spikeinterface\core\job_tools.py", line 304, in run
    results = executor.map(function_wrapper, all_chunks)
  File "E:\envs\si\lib\concurrent\futures\process.py", line 734, in map
    results = super().map(partial(_process_chunk, fn),
  File "E:\envs\si\lib\concurrent\futures\_base.py", line 598, in map
    fs = [self.submit(fn, *args) for args in zip(*iterables)]
  File "E:\envs\si\lib\concurrent\futures\_base.py", line 598, in <listcomp>
    fs = [self.submit(fn, *args) for args in zip(*iterables)]
  File "E:\envs\si\lib\concurrent\futures\process.py", line 705, in submit
    self._adjust_process_count()
  File "E:\envs\si\lib\concurrent\futures\process.py", line 683, in _adjust_process_count
    p.start()
  File "E:\envs\si\lib\multiprocessing\process.py", line 121, in start
    self._popen = self._Popen(self)
  File "E:\envs\si\lib\multiprocessing\context.py", line 327, in _Popen
    return Popen(process_obj)
  File "E:\envs\si\lib\multiprocessing\popen_spawn_win32.py", line 93, in __init__
    reduction.dump(process_obj, to_child)
  File "E:\envs\si\lib\multiprocessing\reduction.py", line 60, in dump
    ForkingPickler(file, protocol).dump(obj)
TypeError: cannot pickle 'weakref' object

This was done on Windows 10, Python 3.10, and spikeinterface was installed from source.

Importantly, the real dataset produces almost the same error, but says TypeError: h5py objects cannot be pickled instead of 'weakref'.

[alejoe91]

Thanks Philipp!

Can confirm that this is Windows-specific:

Can you try to run sorting_well_1 = si.run_sorter('tridesclous2', recording, job_kwargs={'n_jobs' : 1, 'chunk_duration' : '1s', 'verbose' : False})?

Just to confirm that the it's a multiprocessing issue. Also does the save works if you have more than 1 job?

[hornauerp]

Apparently the error only occurs when 'n_jobs' : -1 (which is the default), any other number of jobs runs without error.

[alejoe91]

ahhh! Interesting! We should handle the logic better then when -1 (n_jobs = n_cpus) is used

[yger]

Just to comment, please note that SpykingCircus2 and tridesclous2 are still under development. They are mostly proof of concepts for now that si_based sorters are working, but we need to finish some tests on real GT data before properly validating them...

[alejoe91]

@hornauerp

Should be fixed here: https://github.com/SpikeInterface/spikeinterface/blob/master/spikeinterface/sorters/si_based_sorters/spyking_circus2.py#L49-L51