nerfren icon indicating copy to clipboard operation
nerfren copied to clipboard

Published 20 hours ago • verified publisher icon bennyguo

RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

Open 2694048168 opened this issue 2 years ago • 4 comments

Hi, when I running the script 'nerfren/scripts/nerfren/train_mirror.sh' for training phase,
meeting the Error 'RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)'? Is this Error why? The dimensions of torch.nn.Linear correct? Does anyone encounter this problem and how to solve it？

2694048168 avatar Jan 05 '23 09:01 2694048168

This is probably a torch/cuda version mismatch. Which PyTorch version are you using? How did you install PyTorch?

bennyguo avatar Jan 06 '23 07:01 bennyguo

the CUDA version installed is cuda-11.6, the PyTorch vesion = 1.13.0. And install PyTorch way as official command 'pip3 install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu116'. image

2694048168 avatar Jan 06 '23 10:01 2694048168

I haven't tested on new PyTorch versions. Could you please try to install earlier versions? For example v1.8:

pip3 install torch==1.8.2 torchvision==0.9.2 torchaudio==0.8.2 --extra-index-url https://download.pytorch.org/whl/lts/1.8/cu111

bennyguo avatar Jan 06 '23 10:01 bennyguo

OK, thanks! Let me have a try later.

2694048168 avatar Jan 06 '23 10:01 2694048168