RuntimeError: cublas runtime error : an illegal memory access was encountered

[Jiaoma]

Hi, I met an error when I set deformable_groups>=2 and the size of input is bigger than (2,3,128,128), the error is: RuntimeError: cublas runtime error : resource allocation failed at /opt/conda/conda-bld/pytorch_1544202130060/work/aten/src/THC/THCGeneral.cpp:250 THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1544202130060/work/aten/src/THC/THCCachingHostAllocator.cpp line=265 error=77 : an illegal memory access was encountered

if the input is smaller than (2,3,128,128) or deformable_groups=1, the error disappear. My environment: cuda 10, nvidia-418, pytorch 1.0 cudnn 7, gcc 7.3.0, ubuntu 18.04, dual 2080ti.

[Jiaoma]

By the way, I couldn't find any information about SparseConv2d in Deformable ConvNets v2: More Deformable, Better Results . What is it?

[Jiaoma]

Complete error output: Traceback (most recent call last): File "/home/molijuly/anaconda3/envs/pt1_0py_3_7/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3267, in run_code exec(code_obj, self.user_global_ns, self.user_ns) File "", line 1, in runfile('/home/molijuly/github/SwiftDPP/models/Nets/DUNet.py', wdir='/home/molijuly/github/SwiftDPP/models/Nets') File "/home/molijuly/pycharm-2018.3.2/helpers/pydev/_pydev_bundle/pydev_umd.py", line 197, in runfile pydev_imports.execfile(filename, global_vars, local_vars) # execute the script File "/home/molijuly/pycharm-2018.3.2/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile exec(compile(contents+"\n", file, 'exec'), glob, loc) File "/home/molijuly/github/SwiftDPP/models/Nets/DUNet.py", line 103, in re=net(a) File "/home/molijuly/anaconda3/envs/pt1_0py_3_7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(*input, **kwargs) File "/home/molijuly/github/SwiftDPP/models/Nets/DUNet.py", line 77, in forward enc1 = self.enc1(x) File "/home/molijuly/anaconda3/envs/pt1_0py_3_7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(*input, **kwargs) File "/home/molijuly/github/SwiftDPP/models/Nets/DUNet.py", line 34, in forward return self.encode(x) File "/home/molijuly/anaconda3/envs/pt1_0py_3_7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(*input, **kwargs) File "/home/molijuly/anaconda3/envs/pt1_0py_3_7/lib/python3.7/site-packages/torch/nn/modules/container.py", line 92, in forward input = module(input) File "/home/molijuly/anaconda3/envs/pt1_0py_3_7/lib/python3.7/site-packages/torch/nn/modules/module.py", line 489, in call result = self.forward(*input, **kwargs) File "/home/molijuly/github/dcn/modules/modulated_deform_conv2d.py", line 104, in forward self.im2col_step) File "/home/molijuly/github/dcn/functions/modulated_deform_conv2d_func.py", line 34, in forward ctx.im2col_step) RuntimeError: cublas runtime error : resource allocation failed at /opt/conda/conda-bld/pytorch_1544202130060/work/aten/src/THC/THCGeneral.cpp:250 THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1544202130060/work/aten/src/THC/THCCachingHostAllocator.cpp line=265 error=77 : an illegal memory access was encountered

[Lausannen]

@Jiaoma I met the same problem, have you solved the problem?

[Jiaoma]

@Jiaoma I met the same problem, have you solved the problem?

I use the Deformable ConvNets v2 in mmdetecion, and it works.

[tkkcc]

Finally found a mismatch between offset size and feature size