YOLO-World icon indicating copy to clipboard operation
YOLO-World copied to clipboard

grad_norm为Nan

Open LaplaceSama opened this issue 1 year ago • 3 comments

有没有人出现了跟我一样的问题: 我在训练coco数据集的时候,grad_norm为Nan。我在训练其他数据集时,只要base_lr设置较低,grad_norm就会变成Nan或者Infinity。除此之外,训练coco时,我的loss一直没有动过。我使用的configs/finetune_coco/yolo_world_v2_s_vlpan_bn_2e-4_80e_8gpus_mask-refine_finetune_coco.py中的配置,使用的yolo_world_v2_s_vlpan_bn_2e-4_80e_8gpus_mask-refine_finetune_coco_ep80-492dc329.pth作为预训练路径,配置中没有做任何额外更改。 image

LaplaceSama avatar May 14 '24 01:05 LaplaceSama

same question

Yindong-Zhang avatar May 14 '24 05:05 Yindong-Zhang

如何训练其他的数据集呢

tm924222 avatar May 16 '24 08:05 tm924222

您好,在最开始训练的时候gradients比较大,这些norm是累积的,所以会出现nan,这个是正常的,不会影响结果。

wondervictor avatar May 16 '24 10:05 wondervictor

This issue will be closed since there is no further update related to the main topic. Besides, the error has been fixed already. Thanks for your interest. If you have any questions about YOLO-World in the future, you're welcome to open a new issue.

wondervictor avatar May 26 '24 05:05 wondervictor