Error using gpuArray/subsasgn

[ntotah]

Hi, I just tried Kilosort3 and received an error using gpuArray/subsasgn. I've copied the error below:

Error using gpuArray/subsasgn An unexpected error occurred trying to launch a kernel. The CUDA error was: invalid configuration argument

Error in extract_spikes (line 97) st(5,:) = cF;

Error in main_kilosort3 (line 40) [rez, st3, tF] = extract_spikes(rez);

Here is the info on the GPU: gpuDevice

ans =

CUDADevice with properties:

                  Name: 'Tesla P100-PCIE-16GB'
                 Index: 1
     ComputeCapability: '6.0'
        SupportsDouble: 1
         DriverVersion: 11.1000
        ToolkitVersion: 10
    MaxThreadsPerBlock: 1024
      MaxShmemPerBlock: 49152
    MaxThreadBlockSize: [1024 1024 64]
           MaxGridSize: [2.1475e+09 65535 65535]
             SIMDWidth: 32
           TotalMemory: 1.7072e+10
       AvailableMemory: 1.6689e+10
   MultiprocessorCount: 56
          ClockRateKHz: 1328500
           ComputeMode: 'Default'
  GPUOverlapsTransfers: 1
KernelExecutionTimeout: 0
      CanMapHostMemory: 1
       DeviceSupported: 1
        DeviceSelected: 1

[ntotah]

just an update... I downloaded Kilosort2 and ran the same recording on the same computer and it ran fine.

[holmescharles]

I'm getting this same issue. Surrounding the line 97 from extract_spikes with a try/catch and just "trying again" seems to fix it.

Is there a better fix?

[paul-aparicio]

I also get this same exact error, the try/catch fix worked. If it helps any, I am using Kilosort 3. This error does not occur when I run it on Linux (Ubuntu) with an RTX2070 Super (8gb ram) but does occur when I run it on a server that has a Tesla k40m.

[warnerwarner]

Hi, I've also started getting this, error, I'm guessing there is no update on it yet?

[warnerwarner]

Also to add, if I try the try/catch try again, I end up getting the same error in template_learning (line 71), which I can fix with the try/catch again

[GrimmSnark]

I had the same error and was extremely surprised to see that this fix worked! Thank you

[AthiemoneZero]

I am using Kilosort 3 and received an error using gpuArray/subsasgn like this: "An unexpected error occurred trying to launch a kernal. The CUDA error was:" "invalid configuration argumentERROR: MATLAB error Exit Status: 0*00000001"

this seems not to be common in all of my data, but some of it would happen.

but it fixed when I used try/catch "try again" both surrounding the line 97 from extract_spikes and line 71 in template_learning. Thanks a lot!

BTW, I am using spikeinterface in python which interfaces with Kilosort3 in MATLAB, windows10 os. My GPU is RTX-A4000 (12cores) and running MATLAB with CUDA11.0 toolkits.

In addiction I adjust parameters of spikeinterface.sorters, the n_jobs to -1. This sometimes would happen to error with some of my data, but not all. Im still have not idea with it.