Kilosort runtime error when skip_kilosort_preprocessing=True

[aidapiccato]

I'm getting the following error when running Kilosort2 with the following params: skip_kilosort_preprocessing=True and car=False:

        "[\b> In trackAndSort (line 6)",
        "In runTemplates (line 45)",
        "In learnAndSolve8b (line 24)",
        "In kilosort2_master (line 78)]\b",
        "[\bWarning: An unexpected error occurred during CUDA execution. The CUDA error",
        "was:",
        "CUDA_ERROR_ILLEGAL_ADDRESS]\b",

I'm using cuda=9.1, cudnn=9.1-7.0.5,gcc=5.3.0, matlab=2018b, and spike interface=0.102.3

[zm711]

And you've tried other combinations that have worked? I remember there being some CUDA function issues happening with later versions of Matlab but I think that was 2022a/b +, so 2018 should be fine. Also what OS are you using?

[alejoe91]

@aidapiccato does this happen only when you have skip_preprocessing=True?

[aidapiccato]

@zm711 My operating system is linux. And @alejoe91 yes, this happens only when I have skip_preprocessing=True - any other use of KS2 seems to work fine.

[zm711]

I vaguely remember Aurelien submitting a PR that was never merged into KS that helped deal with regions of relative low voltage that lead to some sort of math error (so his PR was a catch-error or whatever a try-except in matlab is called) to deal with that. Maybe our newest versions return a data structure that has a similar problem? (link to his PR. I looked at the error trace and it has nothing to do with CUDA so I think that is a bad error trace. But the eigenvector issue people were having does occur in runTemplates.

You could try that fix in your local copy of KS2 (unless you're doing docker)? Or you could try to run native matlab version of the code and skip the preprocessing natively to see what happens. Sometimes the error in the Kilosort gui is more informative then what we see in the trace.

[aidapiccato]

Thanks! I made the changes outlined in the PR, but KS still crashes with the same error. I'm running this on a computing cluster, so I'm not sure if I can run the gui, but I'll look into it.

Are users otherwise able to run KS2 with KS preprocessing skipped? For reference, this is my preprocessing pipeline:

    rec = sp.bandpass_filter(recording=rec, freq_min=300, freq_max=10000)
    bad_channel_ids, channel_labels = sp.detect_bad_channels(recording=rec)
    print(f'Bad channels: {bad_channel_ids}')
    rec = sp.interpolate_bad_channels(rec, bad_channel_ids)
    rec = sp.phase_shift(recording=rec)
    rec = sp.common_reference(recording=rec, operator="median", reference="global")
    rec = sp.whiten(rec, int_scale=200)

[zm711]

They have in the past, but I think the issues issue is that python stuff and matlab stuff can very easily get out of sync. So the test we used to have people try would be to copy the matlab file we write out for our wrapper and try to run that script natively in matlab to see if it is a wrapper issue or not.

I'm running this on a computing cluster,

makes sense. You could also try a docker version of KS2 just see what happens, but getting the GPU to engage might also be a struggle. Maybe @alejoe91 has more ideas.?

Depending on the file size I think I have a local copy of KS2 that has worked in the past on a workstation so I could test a file for you.