Scheme 1 align_templates had OverflowError

[HHHHHenry2468]

I was trying to utilize mountainsort5 on our recording file that is about 24GB.

recording = spre.scale(recording, gain=1000, dtype='float32')

Whiten the recording

recording_preprocessed: si.BaseRecording = spre.whiten(recording)

sorting = ms5.sorting_scheme3( recording_preprocessed, sorting_parameters=ms5.Scheme3SortingParameters( block_sorting_parameters=ms5.Scheme2SortingParameters( phase1_detect_channel_radius=50, detect_channel_radius=50, training_duration_sec=60, ), block_duration_sec=60 * 5 ) )

Here are the parameters I set, as our recording probe was the 6-shank ASSY-156-F with RHD2164.

The error occurs when Scheme1 sorting of Scheme2 was aligning the template, and it seems that the avg_offset is too large.

---

OverflowError Traceback (most recent call last) Cell In[13], line 11 8 # Whiten the recording 9 recording_preprocessed: si.BaseRecording = spre.whiten(recording) ---> 11 sorting = ms5.sorting_scheme3( 12 recording_preprocessed, 13 sorting_parameters=ms5.Scheme3SortingParameters( 14 block_sorting_parameters=ms5.Scheme2SortingParameters( 15 phase1_detect_channel_radius=50, 16 detect_channel_radius=50, 17 training_duration_sec=60, 18 ), 19 block_duration_sec=60 * 5 20 ) 21 )

File c:\Users\Lab\miniconda3\envs\kilosort\lib\site-packages\mountainsort5\schemes\sorting_scheme3.py:57, in sorting_scheme3(recording, sorting_parameters) 55 print(f'Processing block {i + 1} of {len(blocks)}...') 56 subrecording = get_block_recording_for_scheme3(recording=recording, start_frame=int(chunk.start) - int(chunk.padding_left), end_frame=int(chunk.end) + int(chunk.padding_right)) ---> 57 result = sorting_scheme2( 58 subrecording, 59 sorting_parameters=sorting_parameters.block_sorting_parameters, 60 return_snippet_classifiers=True, 61 reference_snippet_classifiers=previous_snippet_classifiers, ... --> 264 offsets[k1] = avg_offset 265 if not something_changed: 266 print('Template alignment converged.')

OverflowError: Python int too large to convert to C long

[josephdong1000]

I have had the same issue in align_templates overflowing, though it happens only sporadically for me. My stack trace:

Traceback (most recent call last):
  File "/mnt/isilon/marsh_single_unit/PythonEEG/notebooks/examples/pipeline-war-sox5.py", line 85, in <module>
    sars = ao.compute_spike_analysis(multiprocess_mode='dask')
  File "/mnt/isilon/marsh_single_unit/PythonEEG/pythoneeg/visualization/results.py", line 312, in compute_spike_analysis
    lrec_sorts = dask.compute(*lrec_sorts)
  File "/mnt/isilon/marsh_single_unit/PythonEEG/.venv/lib/python3.10/site-packages/dask/base.py", line 681, in compute
    results = schedule(expr, keys, **kwargs)
  File "/mnt/isilon/marsh_single_unit/PythonEEG/pythoneeg/core/analyze_sort.py", line 140, in _run_sorting
    sorting = sorting_scheme2(
  File "/mnt/isilon/marsh_single_unit/PythonEEG/.venv/lib/python3.10/site-packages/mountainsort5/schemes/sorting_scheme2.py", line 88, in sorting_scheme2
    sorting1 = sorting_scheme1(
  File "/mnt/isilon/marsh_single_unit/PythonEEG/.venv/lib/python3.10/site-packages/mountainsort5/schemes/sorting_scheme1.py", line 136, in sorting_scheme1
    offsets = align_templates(templates)
  File "/mnt/isilon/marsh_single_unit/PythonEEG/.venv/lib/python3.10/site-packages/mountainsort5/schemes/sorting_scheme1.py", line 269, in align_templates
    offsets[k1] = avg_offset
OverflowError: Python integer 4673611590 out of bounds for int32

The error occurs inconsistently every ~100 or so sortings and when I rerun the code it goes away, so I'm not sure if I can give a reproducible example. My pipeline makes a ConcatenatedRecording from several BinaryRecordingExtractors and then splits them to individual channels, then sorts those channels. For preprocessing I do a notch @ 60Hz -> common_ref -> whiten -> bandpass 0.1Hz-100Hz (I'm looking at LFP spiking).

I'm also parallelizing my computation over a cluster with Dask but I don't believe that's the source of the issue.

It would be great to get some info on why/how the error occurs and/or if it is fixable. Thanks!