Hi,
can you please share with us a way to solve this error:
RuntimeError: CUDA out of memory. Tried to allocate 84.00 MiB (GPU 0; 3.82 GiB total capacity; 2.37 GiB already allocated; 76.44 MiB free; 2.52 GiB reserved in total by PyTorch)
First, I though it might be a compatibility issue, even though the message is quite clear that this is not the case so nothing really worked for me.
Now I am having a hard time figuring out how to solve it, I would appreciate some help. Thanks
Here is the full traceback:
Traceback (most recent call last):
File "./tools/train.py", line 109, in
main()
File "./tools/train.py", line 105, in main
logger=logger)
File "/home/husam/qd-3dt/tools/../qd3dt/apis/train.py", line 66, in train_detector
_non_dist_train(model, dataset, cfg, validate=validate)
File "/home/husam/qd-3dt/tools/../qd3dt/apis/train.py", line 270, in _non_dist_train
runner.run(data_loaders, cfg.workflow, cfg.total_epochs)
File "/home/husam/qd-3dt/mmcv/mmcv/runner/runner.py", line 361, in run
epoch_runner(data_loaders[i], **kwargs)
File "/home/husam/qd-3dt/mmcv/mmcv/runner/runner.py", line 264, in train
self.model, data_batch, train_mode=True, **kwargs)
File "/home/husam/qd-3dt/tools/../qd3dt/apis/train.py", line 44, in batch_processor
losses = model(**data)
File "/home/husam/.pyenv/versions/qd_3dt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/husam/.pyenv/versions/qd_3dt/lib/python3.7/site-packages/torch/nn/parallel/data_parallel.py", line 150, in forward
return self.module(*inputs[0], **kwargs[0])
File "/home/husam/.pyenv/versions/qd_3dt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/husam/qd-3dt/tools/../qd3dt/core/fp16/decorators.py", line 49, in new_func
return old_func(*args, **kwargs)
File "/home/husam/qd-3dt/tools/../qd3dt/models/detectrackers/base.py", line 86, in forward
return self.forward_train(img, img_meta, **kwargs)
File "/home/husam/qd-3dt/tools/../qd3dt/models/detectrackers/quasi_dense_3d_sep_uncertainty.py", line 147, in forward_train
x = self.extract_feat(img)
File "/home/husam/qd-3dt/tools/../qd3dt/models/detectrackers/quasi_dense_3d_sep_uncertainty.py", line 118, in extract_feat
x = self.neck(x)
File "/home/husam/.pyenv/versions/qd_3dt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/husam/qd-3dt/tools/../qd3dt/models/necks/dlaup.py", line 158, in forward
x, y = ida(layers[-i - 2:]) # y : aggregation nodes
File "/home/husam/.pyenv/versions/qd_3dt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/husam/qd-3dt/tools/../qd3dt/models/necks/dlaup.py", line 107, in forward
x = node(torch.cat([x, layers[i]], 1))
File "/home/husam/.pyenv/versions/qd_3dt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/husam/qd-3dt/tools/../qd3dt/models/utils/conv_module.py", line 151, in forward
x = self.conv(x)
File "/home/husam/.pyenv/versions/qd_3dt/lib/python3.7/site-packages/torch/nn/modules/module.py", line 532, in call
result = self.forward(*input, **kwargs)
File "/home/husam/.pyenv/versions/qd_3dt/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 345, in forward
return self.conv2d_forward(input, self.weight)
File "/home/husam/.pyenv/versions/qd_3dt/lib/python3.7/site-packages/torch/nn/modules/conv.py", line 342, in conv2d_forward
self.padding, self.dilation, self.groups)
RuntimeError: CUDA out of memory. Tried to allocate 84.00 MiB (GPU 0; 3.82 GiB total capacity; 2.37 GiB already allocated; 76.44 MiB free; 2.52 GiB reserved in total by PyTorch)
SysCV
