When training the dataset, Nan appears

[zhaijs]

When training the dataset, Nan appears

When I started trying to train the COCO2017 dataset, first it was normal, but then there was Nan, which I thought might be the cause of the exploding gradient, but I didn't make any changes to the code. This question has been bothering me for a long time and I look forward to your suggestions.

[zhaijs]

There may be something wrong with OKS loss and I'm trying to use a different loss function

[wanglegoc]

hi @zhaijs Do you solved this problem, would be nice to share some solutions? Thank you

[GeorgeInbelgium]

I meet the same problem, can you share some solutions？

[GeorgeInbelgium]

I find some of my label files are empty. my solution is to modify the label files.

[zhaijs]

I checked the dataset and found no errors. When I calculated kpt_loss_factor, the denominator added 1e-9. kpt_loss_factor = (torch.sum(kpt_mask != 0) + torch.sum(kpt_mask == 0))/(torch.sum(kpt_mask != 0)+1e-9)

[liamsun2019]

I met the same issue. Any solution or workaround for it?