Mystery error while running SC2

[fruce-ki]

SC2 is crashing mid-run, without giving me any indication why.

Preprocessing the recording (bandpass filtering + CMR + whitening)
noise_level (no parallelization): 100%|███████████████████████████████████████████████████████████████████████████████| 20/20 [00:10<00:00,  1.95it/s]
Error running spykingcircus2

It was working fine until I replaced hard-coded parameters in the call with variables, so I could explore the parameter space a bit and find what works for me. So I suspect the problem is a parameter value, but their values in this test run should be the same as the hard-coded ones I was using before.

I would like to recommend that some sanity-checking of parameters is carried out, so that an error message like this can say something useful about where to troubleshoot.

This is my sorter call:

joblist = [
            {
                'sorter_name': 'spykingcircus2', 
                'recording': recordings[row.Well_ID], 
                'output_folder': os.path.join(params.sortedPath, params.experiment, 'spykingcircus2', row.Well_ID),
                'remove_existing_folder' : True,
                'verbose' : True,
                'raise_error' : False,

                'general' : {'ms_before': 2, 'ms_after': 2, 'radius_um': params.GNrad}, 
                'sparsity' : {'method': params.SPmethod, 'amplitude_mode': 'peak_to_peak', 'radius': params.SPrad, 'threshold': params.SPthresh},
                'filtering': {'freq_min': params.BPmin, 'freq_max': params.BPmax, 'ftype': params.BPtype, 'filter_order': params.BPorder, 'margin_ms': params.BPmargin},
                'whitening': {'mode': 'local', 'regularize': False},
                'detection' : {'peak_sign': 'neg', 'detect_threshold': params.DTthresh},
                'selection': {'method': params.SLmethod, 'n_peaks_per_channel': params.SLnppc, 'min_n_peaks': params.SLmnp, 'select_per_channel': False}, 
                'apply_motion_correction' : False,
                'motion_correction': {'preset': 'dredge_fast'},
                'merging': {'similarity_kwargs': {'method': 'cosine', 'support': 'union', 'max_lag_ms': params.MGlag},
                            'correlograms_kwargs': {},
                            'auto_merge': {'min_spikes': params.MGminspikes, 'corr_diff_thresh': params.MGcorrthresh}},
                'clustering' : {'legacy': params.CLlegacy}, 
                'matching' : {'method': 'circus-omp-svd'},
                'apply_preprocessing' : True,
                'matched_filtering' : True,
                'cache_preprocessing': {'mode': 'memory', 'memory_limit': params.resources, 'delete_cache': True},
                'multi_units_only' : False,
                'job_kwargs' : {'n_jobs': params.cores}, ## does not seem to obey global_job_kwargs. Defaults to multithreading and that crashes sc2, so should always be set to 1.
                'debug' : True
            }
            for row in well_pool.itertuples()
            if row.Well_ID in list(recordings.keys())
        ]

Ignore the list comprehension, there is only one recording in this test run.

The values of these parameters for this run are all either values I have previously used successfully, or the default values mentioned in the docs.

Namespace(experiment='DoseRes', row=82, maxrow=82, dataPath='/SCRATCH/mea/data/', preproPath='/SCRATCH/mea/preprocessed/', sortedPath='/SCRATCH/mea/sorted/', cores=1, memory='10G', resources=0.5, tasks=1, sorter='spykingcircus2', BPmin=100, BPmax=3500, BPtype='bessel', BPorder=1, BPmargin=5.0, Wmode='local', GNrad=250, SPmethod='snr', SPrad=20, SPthresh=0.25, DTmethod='locally_exclusive', DTthresh=5, DTsweep=0.1, SLmethod='uniform', SLnppc=5000, SLmnp=100000, MGlag=0.2, MGminspikes=10, MGcorrthresh=0.25, CLlegacy=False, hdf5='/SCRATCH/REFERENCES/maxwell_plugin')

PS. There seems to be disagreement between get_default_sorter_params('spykingcircus2')and get_sorter_params_description('spykingcircus2') about what some of the defaults actually are, for example for the selection dict.

I am using SI version 0.102.0