train.py not running

[2601705041]

my pytorch is 1.10.0 use this: conda install pytorch==1.10.0 torchvision==0.11.0 torchaudio==0.10.0 cudatoolkit=11.3 -c pytorch -c conda-forge because i can only get cpu version use: conda install pytorch==1.9.1 torchvision==0.10.1 torchaudio==0.9.1 cudatoolkit=11.3 -c pytorch -c conda-forge i don't konw why

python train.py Global seed set to 42 Working dir: /home/j222/vid2avatar-main/outputs/Video/parkinglot GPU available: True, used: True TPU available: False, using: 0 TPU cores IPU available: False, using: 0 IPUs Error executing job with overrides: [] Traceback (most recent call last): File "train.py", line 33, in main model = V2AModel(opt) File "/home/j222/vid2avatar-main/code/v2a_model.py", line 30, in init self.model = V2A(opt.model, self.betas_path, self.gender, num_training_frames) File "/home/j222/vid2avatar-main/code/lib/model/v2a.py", line 38, in init self.deformer = SMPLDeformer(betas=betas, gender=self.gender) File "/home/j222/vid2avatar-main/code/lib/model/deformer.py", line 12, in init self.smpl = SMPLServer(gender=gender) File "/home/j222/vid2avatar-main/code/lib/model/smpl.py", line 44, in init output = self.forward(*torch.split(self.param_canonical, [1, 3, 72, 10], dim=1), absolute=True) File "/home/j222/vid2avatar-main/code/lib/model/smpl.py", line 77, in forward v_template=self.v_template) File "/home/j222/vid2avatar-main/code/lib/smpl/body_models.py", line 342, in forward self.lbs_weights, dtype=self.dtype,pose_blend=self.pose_blend) File "/home/j222/vid2avatar-main/code/lib/smpl/lbs.py", line 180, in lbs v_shaped = v_template + blend_shapes(betas, shapedirs) File "/home/j222/vid2avatar-main/code/lib/smpl/lbs.py", line 272, in blend_shapes blend_shape = torch.einsum('bl,mkl->bmk', [betas, shape_disps]) File "/home/j222/anaconda/envs/v2a/lib/python3.7/site-packages/torch/functional.py", line 325, in einsum return einsum(equation, *_operands) File "/home/j222/anaconda/envs/v2a/lib/python3.7/site-packages/torch/functional.py", line 327, in einsum return _VF.einsum(equation, operands) # type: ignore[attr-defined] RuntimeError: CUDA error: CUBLAS_STATUS_INVALID_VALUE when calling cublasSgemmStridedBatched( handle, opa, opb, m, n, k, &alpha, a, lda, stridea, b, ldb, strideb, &beta, c, ldc, stridec, num_batches)

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.