Grad Error!

Open JackAILab opened this issue 2 years ago • 0 comments

When I run the code train.py, some error occurred in "default_runner.py", specifically, loss.backward()

The followings are the detail debug information: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.cuda.FloatTensor [2507, 256]], which is output 0 of ReluBackward0, is at version 1; expected version 0 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

I can find where the grad have been modified? Anyone meet this issue? Thank you for your help!

My environments are:

Jan 20 '23 14:01 JackAILab