RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

[nzh19970106]

Traceback (most recent call last): File "sru_expriment.py", line 181, in train_loss = train(train_loader, criterion, optimizer, model, bsz_train, device) File "sru_expriment.py", line 134, in train output=model(inputs) File "/home/nzh/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(*input, **kwargs) File "sru_expriment.py", line 62, in forward output,c= self.rnns(input) File "/home/nzh/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(*input, **kwargs) File "/home/nzh/anaconda3/lib/python3.7/site-packages/sru/sru_functional.py", line 613, in forward h, c = rnn(prevx, c0[i], mask_pad=mask_pad) File "/home/nzh/anaconda3/lib/python3.7/site-packages/torch/nn/modules/module.py", line 547, in call result = self.forward(*input, **kwargs) File "/home/nzh/anaconda3/lib/python3.7/site-packages/sru/sru_functional.py", line 376, in forward U = self.compute_U(input) File "/home/nzh/anaconda3/lib/python3.7/site-packages/sru/sru_functional.py", line 432, in compute_U U = x.mm(self.weight) RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)

environment：python3.7+pytorch1.2

[dave-rtzr]

I faced with this problem multiple times and found out the version mismatch between CUDA and pytorch was the problem. Try to update your torch and CUDA version.