RuntimeError: Too many grads were not finite

[SSwethaSel0609]

I'm trying to finetune the model using zipformer. I'm facing this issue Traceback (most recent call last): File "finetune.py", line 1532, in main() File "finetune.py", line 1525, in main run(rank=0, world_size=1, args=args) File "finetune.py", line 1403, in run train_one_epoch( File "finetune.py", line 1076, in train_one_epoch scaler.step(optimizer) File "/mnt/efs/swetha/ms_exp/icefall_env/lib/python3.8/site-packages/torch/cuda/amp/grad_scaler.py", line 313, in step return optimizer.step(*args, **kwargs) File "/mnt/efs/swetha/ms_exp/icefall_env/lib/python3.8/site-packages/torch/optim/optimizer.py", line 140, in wrapper out = func(*args, **kwargs) File "/mnt/efs/swetha/ms_exp/icefall_env/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 27, in decorate_context return func(*args, **kwargs) File "/mnt/efs/swetha/ms_exp/icefall/egs/librispeech/ASR/zipformer/optim.py", line 345, in step clipping_scale = self._get_clipping_scale(group, batches) File "/mnt/efs/swetha/ms_exp/icefall/egs/librispeech/ASR/zipformer/optim.py", line 473, in _get_clipping_scale raise RuntimeError("Too many grads were not finite") RuntimeError: Too many grads were not finite

log_error.txt