controlled-text-generation icon indicating copy to clipboard operation
controlled-text-generation copied to clipboard

when running train_discriminator.py ,occuring cudnn RNN backward can only be called in training mode

Open zide05 opened this issue 6 years ago • 5 comments

Traceback (most recent call last): File "train_discriminator.py", line 172, in main() File "train_discriminator.py", line 140, in main loss_G.backward() File "/remote-home/yrchen/anaconda3/envs/py37_cuda8/lib/python3.7/site-packages/torch/tensor.py", line 102, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph) File "/remote-home/yrchen/anaconda3/envs/py37_cuda8/lib/python3.7/site-packages/torch/autograd/init.py", line 90, in backward allow_unreachable=True) # allow_unreachable flag RuntimeError: cudnn RNN backward can only be called in training mode

zide05 avatar Mar 13 '19 02:03 zide05

Hey, you can solve this problem by adding: model.train() to line 152 in file train_discriminator.py

The problem is caused by the function call model.sample_sentence(z, c) calling model.eval() but not calling model.train() afterwards.

pelovett avatar Apr 22 '19 02:04 pelovett

oh,i get this,thank you!

---Original--- From: "Peter Lovett"[email protected] Date: Mon, Apr 22, 2019 10:59 AM To: "wiseodd/controlled-text-generation"[email protected]; Cc: "zide05"[email protected];"Author"[email protected]; Subject: Re: [wiseodd/controlled-text-generation] when running train_discriminator.py ,occuring cudnn RNN backward can only be called in training mode (#24)

Hey, you can solve this problem by adding: model.train() to line 152 in file train_discriminator.py

The problem is caused by the function call model.sample_sentence(z, c) calling model.eval() but not calling model.train() afterwards.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

zide05 avatar Apr 22 '19 08:04 zide05

Hey, you can solve this problem by adding: model.train() to line 152 in file train_discriminator.py

The problem is caused by the function call model.sample_sentence(z, c) calling model.eval() but not calling model.train() afterwards.

this does not solve the issue. same error stays. Any solution??

mahimanzum avatar Oct 02 '19 03:10 mahimanzum

Replace x_gen_attr with x_gen_attr.detach() in train_discriminator.py. i.e in model.forward_encoder_embed and model.forward_discriminator_embed

Akhila-Yerukola avatar Jan 22 '20 19:01 Akhila-Yerukola

Replace x_gen_attr with x_gen_attr.detach() in train_discriminator.py. i.e in model.forward_encoder_embed and model.forward_discriminator_embed

It works, thanks very much!

shizhediao avatar Apr 03 '20 09:04 shizhediao