DeepGlobe-Road-Extraction-Challenge icon indicating copy to clipboard operation
DeepGlobe-Road-Extraction-Challenge copied to clipboard

RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1512946747676/work/torch/lib/THC/generic/THCStorage.cu:58

Open sanersbug opened this issue 6 years ago • 11 comments

when i run python train.py using python virtual environment it shows the error. how to solve the error,anyone solved it? thanks a lot . Another, i have tried to use small batch size,but is doesn't work.

<torch.utils.data.dataloader.DataLoader object at 0x7f3868b7d790> THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1512946747676/work/torch/lib/THC/generic/THCStorage.cu line=58 error=2 : out of memory Traceback (most recent call last): File "train.py", line 47, in train_loss = solver.optimize() File "/mnt/saners-extend/DeepGlobe-Road-Extraction-Challenge/framework.py", line 62, in optimize pred = self.net.forward(self.img) File "/home/anaconda3/envs/road_extract/lib/python2.7/site-packages/torch/nn/parallel/data_parallel.py", line 66, in forward return self.module(*inputs[0], **kwargs[0]) File "/home/anaconda3/envs/road_extract/lib/python2.7/site-packages/torch/nn/modules/module.py", line 325, in call result = self.forward(*input, **kwargs) File "/mnt/saners-extend/DeepGlobe-Road-Extraction-Challenge/networks/dinknet.py", line 176, in forward e3 = self.encoder3(e2) File "/home/anaconda3/envs/road_extract/lib/python2.7/site-packages/torch/nn/modules/module.py", line 325, in call result = self.forward(*input, **kwargs) File "/home/anaconda3/envs/road_extract/lib/python2.7/site-packages/torch/nn/modules/container.py", line 67, in forward input = module(input) File "/home/anaconda3/envs/road_extract/lib/python2.7/site-packages/torch/nn/modules/module.py", line 325, in call result = self.forward(*input, **kwargs) File "/home/anaconda3/envs/road_extract/lib/python2.7/site-packages/torchvision-0.1.9-py2.7.egg/torchvision/models/resnet.py", line 41, in forward File "/home/anaconda3/envs/road_extract/lib/python2.7/site-packages/torch/nn/modules/module.py", line 325, in call result = self.forward(*input, **kwargs) File "/home/anaconda3/envs/road_extract/lib/python2.7/site-packages/torch/nn/modules/conv.py", line 277, in forward self.padding, self.dilation, self.groups) File "/home/anaconda3/envs/road_extract/lib/python2.7/site-packages/torch/nn/functional.py", line 90, in conv2d return f(input, weight, bias) RuntimeError: cuda runtime error (2) : out of memory at /opt/conda/conda-bld/pytorch_1512946747676/work/torch/lib/THC/generic/THCStorage.cu:58

sanersbug avatar Oct 17 '18 03:10 sanersbug