RuntimeError: one of the variables needed for gradient computation has been modified

[yangz0626]

File "/home/pioneer2_1/anaconda3/envs/pointpillars/lib/python3.7/site-packages/mmcv/runner/hooks/optimizer.py", line 19, in after_train_iter runner.outputs['loss'].backward() File "/home/pioneer2_1/anaconda3/envs/pointpillars/lib/python3.7/site-packages/torch/tensor.py", line 118, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph) File "/home/pioneer2_1/anaconda3/envs/pointpillars/lib/python3.7/site-packages/torch/autograd/init.py", line 93, in backward allow_unreachable=True) # allow_unreachable flag RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [11930]], which is output 0 of SelectBackward, is at version 1; expected version 0 instead. Hint: enable anomaly detection to find the operation that failed to compute its gradient, with torch.autograd.set_detect_anomaly(True).

I tried “inplace=False” but failed. How to solve it? Thanks!

[AlphaPlusTT]

i meet the same error when i use the pytorch1.0 and the error disappear when i use the pytorch1.1

[AlphaPlusTT]

@yangz0626 i found one inplace operation in ssd_rotate_head.py:327, maybe you can modify this line and have a try

[dllucas]

@AlphaPlusTT I try to use the pytorch 1.1, but there is something wrong with compiling spconv v1.1. However, I can not find spconv v1.0. Could you please help me?

[zwqnju]

@dllucas I follow the instruction in Det3D/INSTALLATION.md to install spconv-v1.0 ``

[dllucas]

@zwqnju Thank you~ I have also found that.

[yangz0626]

@AlphaPlusTT thanks. It works when I change the pytorch version from 1.3 to 1.1