loss is nan

[thunderbird0902]

我训练自己的数据，刚训练两分钟损失值就变成nan，请问这是什么原因导致的？

[hkzhang95]

Sorry for the late reply, I am kind of busy recently.

There are a lot of possibilities for this case and I cannot find the reason just by loss is nan. Here are some common reasons:

• Noisy data
  • Images containing no objects?
  • Noisy annotations?
• Unreasonable settings
  • Data settings. Categories? Mappings?
  • Hyperparameters. Anchor settings? Mini-batch size? Dynamic R-CNN hyperparameters?
• etc.

Maybe you can provide more detailed information (everything might be helpful). For example:

• If you use the original Faster R-CNN with your data, is your loss normal?
• Can you make sure that the settings are correct?
• What kind of data you are using? Does it have some special properties which require some considerate hyperparameters?