RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED

[Bongnah]

python main.py --title HOZ --model HOZ --workers 12 --gpu-ids 0 python full_eval.py --title HOZ --model HOZ --results-json HOZ.json --gpu-ids 0

When I ran the above two codes for training and evaluation, I got the following error.

File "/home//miniconda3/envs/ng/lib/python3.6/site-packages/torch/nn/functional.py", line 1369, in linear ret = torch.addmm(bias, input, weight.t()) RuntimeError: CUDA error: CUBLAS_STATUS_EXECUTION_FAILED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)

I wonder how to solve these issues