SinGAN icon indicating copy to clipboard operation
SinGAN copied to clipboard

RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation

Open fatimahgit opened this issue 3 years ago • 4 comments

can anyone please help me fix this error, I trained the model by running:

python main_train.py --input_name <input_file_name>

and got this error: RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3, 32, 3, 3]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

Full error message:

python main_train.py --not_cuda --input_name 33039_LR.png Random Seed: 1310 GeneratorConcatSkip2CleanAdd( (head): ConvBlock( .......... )

..\aten\src\ATen\native\TensorFactories.cpp:361: UserWarning: Deprecation warning: In a future PyTorch release torch.full will no longer return tensors of floating dtype by default. Instead, a bool fill_value will return a tensor of torch.bool dtype, and an integral fill_value will return a tensor of torch.long dtype. Set the optional dtype or out arguments to suppress this warning. Warning: Error detected in MkldnnConvolutionBackward. No forward pass information available. Enable detect anomaly during forward pass for more information. (print_stack at ..\torch\csrc\autograd\python_anomaly_mode.cpp:42) Traceback (most recent call last): File "main_train.py", line 29, in train(opt, Gs, Zs, reals, NoiseAmp) File "C:\Users\z5029775\Downloads\SinGAN-master\SinGAN\training.py", line 39, in train z_curr,in_s,G_curr = train_single_scale(D_curr,G_curr,reals,Gs,Zs,in_s,NoiseAmp,opt) File "C:\Users\z5029775\Downloads\SinGAN-master\SinGAN\training.py", line 179, in train_single_scale errG.backward(retain_graph=True) File "C:\Users\z5029775\AppData\Local\Continuum\anaconda3\envs\fatimah\lib\site-packages\torch\tensor.py", line 198, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph) File "C:\Users\z5029775\AppData\Local\Continuum\anaconda3\envs\fatimah\lib\site-packages\torch\autograd_init_.py", line 98, in backward Variable._execution_engine.run_backward( RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation: [torch.FloatTensor [3, 32, 3, 3]] is at version 2; expected version 1 instead. Hint: the backtrace further above shows the operation that failed to compute its gradient. The variable in question was changed in there or anywhere later. Good luck!

fatimahgit avatar Aug 06 '20 06:08 fatimahgit

I have the same question in pytorch 1.6.0. And I change to another environment with pytorch 1.3.1, the question is fixed. I don't know the reasons, but it works. Hope this method can help you.

lioo717 avatar Aug 13 '20 12:08 lioo717

Please use torch==1.4.0 torchvision==0.5.0.

tamarott avatar Aug 17 '20 10:08 tamarott

You should modify the requirements file to reflect this. Thanks!

andrew-jeremy avatar Nov 17 '20 00:11 andrew-jeremy

任何人都可以帮我修复这个错误,我通过运行来训练模型:

python main_train.py --input_name <input_file_name>

并收到此错误:RuntimeError:梯度计算所需的变量之一已被就地操作修改:[torch.FloatTensor [3, 32, 3, 3]] 版本为 2;而是预期的版本 1。提示:上面的回溯显示未能计算其梯度的操作。有问题的变量在那里或以后的任何地方都发生了变化。祝你好运!

完整的错误消息:

python main_train.py --not_cuda --input_name 33039_LR.png 随机种子:1310 GeneratorConcatSkip2CleanAdd( (head): ConvBlock( .......... )

..\aten\src\ATen\native\TensorFactories.cpp:361:用户警告:弃用警告:在未来的 PyTorch 版本中,torch.full 将不再默认返回浮动 dtype 的张量。相反,bool fill_value将返回 torch.bool dtype 的张量,而积分fill_value将返回 torch.long dtype 的张量。设置 optional 或 arguments 以禁止显示此警告。警告:在 MkldnnConvolutionBackward 中检测到错误。没有可用的正向传递信息。启用在前向传递期间检测异常,以获取更多信息。(print_stack..\torch\csrc\autograd\python_anomaly_mode.cpp:42) 回溯(最近一次调用最后一次): 文件“main_train.py”,第 29 行,在 train(opt, Gs, Zs, reals, NoiseAmp) 文件“C:\Users\z5029775\Downloads\SinGAN-master\SinGAN\training.py”,第 39 行,在列车 z_curr,in_s,G_curr = train_single_scale(D_curr,G_curr,reals,Gs,Zs,in_s,NoiseAmp,opt) 文件“C:\Users\z5029775\Downloads\SinGAN-master\SinGAN\training.py“,第 179 行,train_single_scale errG.backward(retain_graph=True) 文件”C:\Users\z5029775\AppData\Local\Continuum\anaconda3\envs\fatimah\lib\site-packages\torch\tensor.py“,第 198 行,在 backward torch.autograd.backward(self, gradient, retain_graph, create_graph) 文件”C:\Users\z5029775\AppData\Local\Continuum\anaconda3\envs\fatimah\lib\site-packages\torch\autograd__init__.py“,第 98 行,在 backward Variable._execution_engine.run_backward( RuntimeError:梯度计算所需的变量之一已被就地操作修改:[torch.FloatTensor [3, 32, 3, 3]] 版本为 2;而是预期的版本 1。提示:上面的回溯显示未能计算其梯度的操作。有问题的变量在那里或以后的任何地方都发生了变化。祝你好运!dtype``out

can you solve it ? if ok, help me pleae!

YL-Wang1 avatar Nov 30 '23 02:11 YL-Wang1