Kilosort running Error

[paul-aparicio]

Hello, I am having two problems while running kilosort3 and I am not really sure how to utilize the error messages to figure out what is wrong with my data, any help would be very appreciated. I am running kilosort 3 on matlab 2020b on ubuntu 18.04.

(1) The first error I get is the following: Error Running Kilosort! An unexpected error occurred trying to launch a kernel. The CUDA error was: invalid configuration argument. (2) The second error is: Error saving data to phy! Reference to non-existent field 'Wphy'

The output to the command window is:

Channel-whitening matrix computed. Time 8s. Loading raw data and applying filters... Time 717s. Finished preprocessing 2545 batches. pitch is 400 um 0.09 sec, 1 batches, 10000 spikes 8.75 sec, 101 batches, 994143 spikes 27.28 sec, 201 batches, 1922768 spikes 35.73 sec, 301 batches, 2814145 spikes 44.20 sec, 401 batches, 3644829 spikes 52.46 sec, 501 batches, 4537196 spikes 60.97 sec, 601 batches, 5462584 spikes 69.40 sec, 701 batches, 6422091 spikes 78.44 sec, 801 batches, 7422091 spikes 86.92 sec, 901 batches, 8418679 spikes 95.27 sec, 1001 batches, 9351200 spikes 103.60 sec, 1101 batches, 10211299 spikes 112.26 sec, 1201 batches, 11123574 spikes 120.66 sec, 1301 batches, 11960540 spikes 129.02 sec, 1401 batches, 12862619 spikes 137.50 sec, 1501 batches, 13794744 spikes 145.69 sec, 1601 batches, 14701479 spikes 153.98 sec, 1701 batches, 15596164 spikes 162.27 sec, 1801 batches, 16528884 spikes 170.67 sec, 1901 batches, 17528884 spikes 179.41 sec, 2001 batches, 18475806 spikes 187.53 sec, 2101 batches, 19409601 spikes 189.55 sec, 2201 batches, 20284130 spikes 191.59 sec, 2301 batches, 21215634 spikes 193.86 sec, 2401 batches, 22115751 spikes 195.72 sec, 2501 batches, 22362084 spikes 196.53 sec, 2545 batches, 22362084 spikes time 541.99, Shifted up/down 2545 batches. Time 546s. Optimizing templates ...

[marius10p]

Looks like too many spikes. What kind of probe is this? Is the pitch really 400um? Maybe there is a configuration error, does the data look right in the GUI ?

[paul-aparicio]

Thank you for your reply. The probe is a 96 channel utah array. I have the probe configured as an imaginary line where the distance between probes is set to 400um. The data looks fine in the gui, and I have run several smaller datasets (2-3gb) through successfully with the same configuration, but larger datafiles (~32gb) produce the same error.

[marius10p]

What gpu are you using? Do you get the same error in Kilosort 2.5?

[paul-aparicio]

Hi, I am using an RTX2070 Super (8gb memory). I have tried the dataset using 2.5 and 3.0, both return the same error.

[marius10p]

Send a screenshot of the data from the GUI please. Both the raw and filtered views.

On Mon, 3 May 2021 at 08:03, paul-aparicio @.***> wrote:

Hi, I am using an RTX2070 Super (8gb memory). I have tried the dataset using 2.5 and 3.0, both return the same error.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/MouseLand/Kilosort/issues/384#issuecomment-831215564, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA6AYDQZV53UK6TIEDIWKZTTL2GK5ANCNFSM43RZ22FQ .

[paul-aparicio]

Here are a few screen shots of two time points (filtered and raw):

With the scale increased to better show the filtered data

[paul-aparicio]

Hi @marius10p , were these images ok?

[aboharbf]

Also just ran into error: ' Error Running Kilosort! An unexpected error occurred trying to launch a kernel. The CUDA error was: invalid configuration argument.'

The line specifically is in extract_spikes:

%     st(5,:) = cF; % Invalid configuration argument CUDA Error
    st = [st; cF]; % Fixes it

as the code shows, I can get around it by doing a stacking rather than direct assignment. The previous slice of this data (recording on the same day) does fine, and the st is 5 * 77, where as it fails when st is 5 * ~230. Running on a GTX 1070, MATLAB 2018a, CUDA 9.0.

Ran into this same problem using a RTX 3080, MATLAB 2021a, and CUDA Toolkit 11.0. I put the block below:

    try
        st(5,:) = cF;
    catch 
        st = [st; cF];
    end

and it seems to allow it to get through some of these errors. Just replacing the first statement with the 2nd seems to produce more errors. Seems the files for which one works and the other doesn't is random.