RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling `cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc)`

[jasonwjw]

Test error: File "/gpfs/work/mec/junweiwu20/SparseTT-main/videoanalyst/model/backbone/backbone_impl/swin_transformer.py", line 94, in forward qkv = self.qkv(x).reshape(B, N, 3, self.num_heads, File "/gpfs/work/mec/junweiwu20/anaconda3/envs/SparseTT/lib/python3.7/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl return forward_call(*input, **kwargs) File "/gpfs/work/mec/junweiwu20/anaconda3/envs/SparseTT/lib/python3.7/site-packages/torch/nn/modules/linear.py", line 103, in forward return F.linear(input, self.weight, self.bias) File "/gpfs/work/mec/junweiwu20/anaconda3/envs/SparseTT/lib/python3.7/site-packages/torch/nn/functional.py", line 1848, in linear return torch._C._nn.linear(input, weight, bias) RuntimeError: CUDA error: CUBLAS_STATUS_NOT_SUPPORTED when calling cublasSgemm( handle, opa, opb, m, n, k, &alpha, a, lda, b, ldb, &beta, c, ldc) Process Process-2:

cuda version: nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2021 NVIDIA Corporation Built on Mon_May__3_19:15:13_PDT_2021 Cuda compilation tools, release 11.3, V11.3.109 Build cuda_11.3.r11.3/compiler.29920130_0

pytorch version: python 3.7.12 hf930737_100_cpython conda-forge python-dateutil 2.9.0.post0 pypi_0 pypi python_abi 3.7 4_cp37m conda-forge pytorch 1.10.0 py3.7_cuda11.3_cudnn8.2.0_0 pytorch pytorch-mutex 1.0 cuda pytorch pytz 2024.1 pypi_0 pypi

This error is very strange, I googled the problem, most of the responses are the inconsistent version of pytorch and cuda or the batchsize, it seems not correct in this case, Could someone give me some advice?