style-based-gan-pytorch icon indicating copy to clipboard operation
style-based-gan-pytorch copied to clipboard

Published 20 hours ago • verified publisher icon rosinality

CUDNN_STATUS_NOT_SUPPORTED

Open edebrouwer opened this issue 5 years ago • 8 comments

Hi, I'm having this error while trying to train the model on CelebA dataset :

Traceback (most recent call last): File "train.py", line 347, in train(args, dataset, generator, discriminator, device) File "train.py", line 167, in train fake_predict.backward() File "/home/edward/anaconda3/lib/python3.6/site-packages/torch/tensor.py", line 195, in backward torch.autograd.backward(self, gradient, retain_graph, create_graph) File "/home/edward/anaconda3/lib/python3.6/site-packages/torch/autograd/init.py", line 99, in backward allow_unreachable=True) # allow_unreachable flag RuntimeError: cuDNN error: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input.

I'm running Python 3.6.4 with torch 1.4.0. Nvidia-driver 418.56

Any idea where I could look for some fix ?

Thanks so much !

Edward

edebrouwer avatar Feb 14 '20 18:02 edebrouwer

Could you give me the error logs with CUDA_LAUNCH_BLOCKING=1?

rosinality avatar Feb 14 '20 23:02 rosinality

I have exactly the same error logs with CUDA_LAUNCH_BLOKCING=1

edebrouwer avatar Feb 16 '20 20:02 edebrouwer

Could you check output dimensions? Sometimes this kind of error happens when tensor is too large.

rosinality avatar Feb 17 '20 01:02 rosinality

I tried with smaller batch sizes and it still fails on the lowest resolution.

On 17 Feb 2020, at 02:10, Kim Seonghyeon [email protected] wrote:

Could you check output dimensions? Sometimes this kind of error happens when tensor is too large.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/rosinality/style-based-gan-pytorch/issues/83?email_source=notifications&email_token=AFM3FN6C6JLQOOJRO3GZPF3RDHPZPA5CNFSM4KVNFF3KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEL4YY5Q#issuecomment-586779766, or unsubscribe https://github.com/notifications/unsubscribe-auth/AFM3FNZ7LDXZNCZK4HD65XDRDHPZPANCNFSM4KVNFF3A.

edebrouwer avatar Feb 17 '20 13:02 edebrouwer

Sorry but it is hard to know why...

rosinality avatar Feb 17 '20 14:02 rosinality

@edebrouwer try to upgrade torch 1.4.0 to nightly, it helped me. (cuda 10.0, torch 1.5.0.dev20200313+cu100)

Dimbl4 avatar Apr 10 '20 22:04 Dimbl4

Same issue. Did anyone fix it?

KelestZ avatar Apr 12 '20 16:04 KelestZ

I had same issue in here, so I downgrade the torch version. For me, torch 1.2.0 + cuda 10.0 works well (link)

seongeunso avatar Jul 23 '20 03:07 seongeunso