Kilosort
Kilosort copied to clipboard
MouseLand
CUDA_ERROR_ILLEGAL_ADDRESS during preprocessing in standalone_detector
While running the newest Kilosort code, I reliably run into the following problem:
Error using gpuArray/gather
An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
Error in gpuArray/subsindex (line 17)
idx = gather( idx ) - 1;
Error in standalone_detector (line 68)
ys = rez.yc(iC);
Error in datashift2 (line 58)
[st3, rez] = standalone_detector(rez, spkTh);
Error in ksGUI/runPreproc (line 726)
obj.rez = datashift2(obj.rez, 1);
Error in ksGUI>@(~,~)obj.runPreproc() (line 331)
'Callback', @(~,~)obj.runPreproc());
Error while evaluating UIControl Callback.
When trying to debug the problem, I was able to trace it back to the spikedetector3 MEX-file in the function standalone_detector, more specifically the for loop, starting in row 56. The first iteration seems to work fine, but during the second, spikedetector3 throws following warnings:
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 64)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 64)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 64)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 64)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 64)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 64)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Before that, I am able to create gpuArrays, but precisely after that line I am not anymore.
Any ideas on how to solve the problem? I already tried to follow @nmtimme's recommendation from #257 (adding a line after getClosestChannels2), but unfortunately without success. The mexGPUall compile works fine, altough I had to disable "enableStableMode", in order to not run into errors with mexcuda -largeArrayDims -dynamic -DENABLE_STABLEMODE mexMPnu8.cu
To my equipment: OS: Win10 64x Graphics: Nvidia GeForce GTX1060 6GB Matlab version: 2020b CUDA ToolkitVersion: 10.2 (as recommended by Matlab), all drivers up to date Compiler (C++): Microsoft Visual C++ 2017 (but I also tried with 2015 and 2019)
CUDA errors are thrown asynchronously, so the code is probably failing in the first call to spikedetector3. What errors did you get in stable mode? Also, just before that first call to spikedetector3, can you put a debug point, and inspect dataRAW to see if it looks reasonable? Did you look at the raw data in the GUI, does it look fine?
Thanks for your quick response! The mex file compilation error, was the following:
Error using mex
fatbinary fatal : '98_device.compute_35.cubin' is not in 'keyword=value' format
mexMPnu8.obj
c_mexapi_version.obj
Error in mexcuda (line 166)
[varargout{1:nargout}] = mex(mexArguments{:});
Error in mexGPUall (line 18)
mexcuda -largeArrayDims -dynamic -DENABLE_STABLEMODE mexMPnu8.cu
Error in run (line 91)
evalin('caller', strcat(script, ';'));
However I actually managed to get around that problem with the great help from issue #220 (probably even worth putting into the general kilosort docs). Nonetheless, the error described above persisted.
The dataRAW traces look reasonable and the channels in the GUI also look fine. Maybe I also have to add, that I was working with kilosort2 already for a long time, with the same tetrode arrays and channel maps (or do I have to map them differently in this new kilosort version?). So it probably has something to do with the new kilosort3 update.
Would verbosing the error help or is that useless, since CUDA errors are usually a bit meaningless?
Channel maps are the same. I don't really know how to get verbose information from CUDA errors, do you know of a way? That mex function is new since Kilosort2.5 which has been up for several months, and I don't think anyone's reported similar problems yet. Can you try to sort a different segment of your data, for example with ops.trange = [100 200] in seconds?
Yeah, CUDA errors are tricky. I heard about CUDA-GDB debugging, but I'm not that familiar with the CUDA environment. Unfortunately changing the trange didn't alter the error message...
Could it also be an issue with the newest Matlab version? @ffredes Are you using the same environment (i.e. Matlab 2020b, same Toolkit, Compiler etc.)?
I have updated all the drivers. I have matlab 2018a, CUDA 9.0, visual studio 2013. I have tried in two different machines, same problem.
So, I just updated the graphics driver from 456.71 to 461.40. Actually I was just about to post, that preprocessing works fine now, as it did iterate through nearly all of the 421 batches. But after "725.10 sec, 401 batches, 1254744 spikes" the same error with the same preceeding warning showed up. Any tips on how to further enhance CUDA stability? Is there a known compiler problem of visual studio? I inferred from #75 that MinGW64 might sometimes work more stable.
Are you sure the error is in the same place, inside that loop? Did you recompile all of the mex scripts? Is this visual studio community? That 2013 version always seemed to work fine. I never tried MinGW64.
The error seems to be due to the same cause, although the warning message before the error call is slightly prolonged now:
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 64)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 64)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 64)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 64)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 64)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 64)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 64)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 64)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 64)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 64)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 64)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 10)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 10)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 10)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 10)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 10)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 10)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 10)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In datashift2 (line 3)
In ksGUI/runPreproc (line 726)
In ksGUI>@(~,~)obj.runPreproc() (line 331)
Error using gpuArray/gather
An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
Error in gpuArray/subsindex (line 17)
idx = gather( idx ) - 1;
Error in standalone_detector (line 68)
ys = rez.yc(iC);
Error in datashift2 (line 58)
[st3, rez] = standalone_detector(rez, spkTh);
Error in ksGUI/runPreproc (line 726)
obj.rez = datashift2(obj.rez, 1);
Error in ksGUI>@(~,~)obj.runPreproc() (line 331)
'Callback', @(~,~)obj.runPreproc());
Error while evaluating UIControl Callback.
I did recompile the mex scripts and I am using the community version of visual studio. I will try with the 2013 version.
Hi guys! I have exactly the same issue reported originally by FilippoHei:
Time 0s. Computing whitening matrix..
Getting channel whitening matrix...
Channel-whitening matrix computed.
Time 2s. Loading raw data and applying filters...
Time 74s. Finished preprocessing 99 batches.
vertical pitch size is 28
horizontal pitch size is 28
Columns 1 through 9
0 14 28 42 56 70 84 98 112
Columns 10 through 18
126 140 154 168 182 196 210 224 238
Columns 19 through 27
252 266 280 294 308 322 336 350 364
Columns 28 through 36
378 392 406 420 434 448 462 476 490
Columns 37 through 45
504 518 532 546 560 574 588 602 616
Columns 46 through 54
630 644 658 672 686 700 714 728 742
Columns 55 through 63
756 770 784 798 812 826 840 854 868
Columns 64 through 72
882 896 910 924 938 952 966 980 994
Columns 73 through 81
1008 1022 1036 1050 1064 1078 1092 1106 1120
Columns 82 through 90
1134 1148 1162 1176 1190 1204 1218 1232 1246
Columns 91 through 99
1260 1274 1288 1302 1316 1330 1344 1358 1372
Columns 100 through 108
1386 1400 1414 1428 1442 1456 1470 1484 1498
Columns 109 through 117
1512 1526 1540 1554 1568 1582 1596 1610 1624
Columns 118 through 123
1638 1652 1666 1680 1694 1708
713
1.32 sec, 1 batches, 1927 spikes
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 63)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI/runAll (line 627)
In ksGUI>@(~,~)obj.runAll() (line 319)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 63)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI/runAll (line 627)
In ksGUI>@(~,~)obj.runAll() (line 319)
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In standalone_detector (line 63)
In datashift2 (line 58)
In ksGUI/runPreproc (line 726)
In ksGUI/runAll (line 627)
In ksGUI>@(~,~)obj.runAll() (line 319)
...
[MANY WARNING LIKE THIS]
...
Warning: An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
> In datashift2 (line 3)
In ksGUI/runPreproc (line 726)
In ksGUI/runAll (line 627)
In ksGUI>@(~,~)obj.runAll() (line 319)
Error using gpuArray/gather
An unexpected error occurred during CUDA execution. The CUDA error was:
CUDA_ERROR_ILLEGAL_ADDRESS
Error in gpuArray/subsindex (line 17)
idx = gather( idx ) - 1;
Error in standalone_detector (line 67)
ys = rez.yc(iC);
Error in datashift2 (line 58)
[st3, rez] = standalone_detector(rez, spkTh);
Error in ksGUI/runPreproc (line 726)
obj.rez = datashift2(obj.rez, 1);
Error in ksGUI/runAll (line 627)
obj.runPreproc;
Error in ksGUI>@(~,~)obj.runAll() (line 319)
'Callback', @(~,~)obj.runAll());
Error while evaluating UIControl Callback.
My equipment: OS: Win10 64x RAM: 128GB Matlab version: 2020b CUDA ToolkitVersion: 10.2 (as recommended by Matlab), all drivers up to date Compiler (C++): Microsoft Visual C++ 2017
CUDADevice with properties:
Name: 'GeForce RTX 2080 Ti'
Index: 1
ComputeCapability: '7.5'
SupportsDouble: 1
DriverVersion: 11.2000
ToolkitVersion: 10.2000
MaxThreadsPerBlock: 1024
MaxShmemPerBlock: 49152
MaxThreadBlockSize: [1024 1024 64]
MaxGridSize: [2.1475e+09 65535 65535]
SIMDWidth: 32
TotalMemory: 1.1811e+10
AvailableMemory: 9.5415e+09
MultiprocessorCount: 68
ClockRateKHz: 1560000
ComputeMode: 'Default'
GPUOverlapsTransfers: 1
KernelExecutionTimeout: 1
CanMapHostMemory: 1
DeviceSupported: 1
DeviceSelected: 1
I also tried to recompile the mex scripts and with Matlab2019b but the isssue still persist.
If someone can send me a minimal example that replicates the error (ie 10sec of data), that would be great.
On Wed, Feb 10, 2021, 6:11 AM nets3lab [email protected] wrote:
Hi guys! I have exactly the same issue reported originally by FilippoHei:
Time 0s. Computing whitening matrix.. Getting channel whitening matrix... Channel-whitening matrix computed. Time 2s. Loading raw data and applying filters... Time 74s. Finished preprocessing 99 batches. vertical pitch size is 28 horizontal pitch size is 28 Columns 1 through 9
0 14 28 42 56 70 84 98 112Columns 10 through 18
126 140 154 168 182 196 210 224 238Columns 19 through 27
252 266 280 294 308 322 336 350 364Columns 28 through 36
378 392 406 420 434 448 462 476 490Columns 37 through 45
504 518 532 546 560 574 588 602 616Columns 46 through 54
630 644 658 672 686 700 714 728 742Columns 55 through 63
756 770 784 798 812 826 840 854 868Columns 64 through 72
882 896 910 924 938 952 966 980 994Columns 73 through 81
1008 1022 1036 1050 1064 1078 1092 1106 1120Columns 82 through 90
1134 1148 1162 1176 1190 1204 1218 1232 1246Columns 91 through 99
1260 1274 1288 1302 1316 1330 1344 1358 1372Columns 100 through 108
1386 1400 1414 1428 1442 1456 1470 1484 1498Columns 109 through 117
1512 1526 1540 1554 1568 1582 1596 1610 1624Columns 118 through 123
1638 1652 1666 1680 1694 1708713
1.32 sec, 1 batches, 1927 spikes Warning: An unexpected error occurred during CUDA execution. The CUDA error was: CUDA_ERROR_ILLEGAL_ADDRESS
In standalone_detector (line 63) In datashift2 (line 58) In ksGUI/runPreproc (line 726) In ksGUI/runAll (line 627) In ksGUI>@(~,~)obj.runAll() (line 319) Warning: An unexpected error occurred during CUDA execution. The CUDA error was: CUDA_ERROR_ILLEGAL_ADDRESS In standalone_detector (line 63) In datashift2 (line 58) In ksGUI/runPreproc (line 726) In ksGUI/runAll (line 627) In ksGUI>@(~,~)obj.runAll() (line 319) Warning: An unexpected error occurred during CUDA execution. The CUDA error was: CUDA_ERROR_ILLEGAL_ADDRESS In standalone_detector (line 63) In datashift2 (line 58) In ksGUI/runPreproc (line 726) In ksGUI/runAll (line 627) In ksGUI>@(~,~)obj.runAll() (line 319) ... [MANY WARNING LIKE THIS] ... Warning: An unexpected error occurred during CUDA execution. The CUDA error was: CUDA_ERROR_ILLEGAL_ADDRESS In datashift2 (line 3) In ksGUI/runPreproc (line 726) In ksGUI/runAll (line 627) In ksGUI>@(~,~)obj.runAll() (line 319) Error using gpuArray/gather An unexpected error occurred during CUDA execution. The CUDA error was: CUDA_ERROR_ILLEGAL_ADDRESS
Error in gpuArray/subsindex (line 17) idx = gather( idx ) - 1;
Error in standalone_detector (line 67) ys = rez.yc(iC);
Error in datashift2 (line 58) [st3, rez] = standalone_detector(rez, spkTh);
Error in ksGUI/runPreproc (line 726) obj.rez = datashift2(obj.rez, 1);
Error in ksGUI/runAll (line 627) obj.runPreproc;
Error in ksGUI>@(~,~)obj.runAll() (line 319) 'Callback', @(~,~)obj.runAll());
Error while evaluating UIControl Callback.
My equipment: OS: Win10 64x Graphics: Nvidia GeForce GTX1080 Ti Matlab version: 2020b CUDA ToolkitVersion: 10.2 (as recommended by Matlab), all drivers up to date Compiler (C++): Microsoft Visual C++ 2017
I also tried to recompile the mex scripts and with Matlab2019b but the isssue still persist.
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/MouseLand/Kilosort/issues/333#issuecomment-776634209, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA6AYDTGTMGHP2C55KHRQRLS6JSVDANCNFSM4XJMOZWQ .
Of course Marius! Could you give me an e-mail where I can send you a private link to download the dataset?
Best
Find my email address from a recent paper please (ie Cellpose).
On Wed, Feb 10, 2021, 7:10 AM nets3lab [email protected] wrote:
Of course Marius! Could you give me an e-mail where I can send you a private link to download the dataset?
Best
— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/MouseLand/Kilosort/issues/333#issuecomment-776664265, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA6AYDQG3UJO2QN3FBATS73S6JZSTANCNFSM4XJMOZWQ .
Hi there, I have solved the issue by increasing the "CUDA_CACHE_MAXSIZE " You have to create an environment variable CUDA_CACHE_MAXSIZE with this size "1073741824 " Best
For what it's worth we're having the same issue. Tried creating the CUDA_CACHE_MAXSIZE variable as suggested, but still got the same error.
We just tried running this again and came up with these same errors. Updated the graphics drivers, upgraded to Visual Studio Community 2017, recompiled the MEX files (no issues, though we didn't have an issue during the initial compiling either). Here's our setup:
OS: Win10 64x RAM: 64GB Matlab version: 2019a GPU: Nvidia GeForce RTX 2070 Super CUDA ToolkitVersion: 10.0 Compiler (C++): Microsoft Visual C++ 2017 (community, previously using 2013)
Has there been any luck resolving this?
Hi all, Has there been any success in solving this? I'm running into this same issue with KS-2.5.
I've tried increasing the CUDA_CACHE_MAXSIZE to 1GB, set TdrLevel = 0, recompiled the MEX files, updated the graphics drivers and am still getting the error: CUDA_ERROR_ILLEGAL_ADDRESS
In standalone_detector (line 11) In datashift2 (line 31) In ksGUI/runPreproc (line 726) In ksGUI>@(~,~)obj.runPreproc() (line 331)
We're using: OS: Win10 64x RAM: 32GB Matlab version: 2020b GPU: Nvidia Quadro p1000 CUDA ToolkitVersion: 10.2 (as per recommended w/Matlab2020b) Compiler (C++): Microsoft Visual C++ 2017 (community, 2013 version is not supported by Matlab2020b)
I was able to get it to sort successfully if I changed N blocks for registration = 0. Any values > 0 causes it to break in the standalone_detector with a CUDA_ERROR_ILLEGAL_ADDRESS.
Any suggestions to be able to use drift correction? Our plans involve chronically implanted NP probes and sessions ~1hr so drift corrections would be helpful.
I faced the same error when I used the channel map created by myself. I used the Neuropixels 1.0 and allocated the odd number channels to shank 1 and even to shank 2. In that case, I found that datashift2.m assigned 5 to dminx, and it caused the misidentification of the number of x columns because the actual span between x columns was not 5 micrometers (it should be 32 in my case). So I changed the script of datashift2.m, as shown below, and I could run Kilosort successfully.
dminx = median(diff(unique(rez.xc)));
% yunq = unique(rez.yc); % Pick up unique values of ycoords
% mxc = zeros(numel(yunq), 1);
% for j = 1:numel(yunq)
% xc = rez.xc(rez.yc==yunq(j)); % Set of xcoords corresponding to each unique ycoords value
% if numel(xc)>1
% mxc(j) = median(diff(sort(xc))); % If there are more than 1 xcoords value for a certain ycoords value, take median of them
% end
% end
% dminx = max(5, median(mxc));
