Detectron.pytorch icon indicating copy to clipboard operation
Detectron.pytorch copied to clipboard

Handle NaN loss

Open MFattouh opened this issue 7 years ago • 2 comments

To avoid division by zero or passing zeros or negative values to log in bbox_transform_inv I replace those values with cfg.EPS. In addition, the NaN mask loss which rises after a negative area calculation is handled by discarding the entire batch.

MFattouh avatar Jul 20 '18 09:07 MFattouh

This might be a tempting fix, but in my opinion if you have a NaN loss somewhere, then there is a serious problem in your network which deserves a deeper investigation.

There is probably a good reason why FAIR did not implement it that way in the first place.

heinzermch avatar Jul 23 '18 23:07 heinzermch

well, you're absolutely right. NaN should not "normally" appear. Otherwise, A deeper investigation is required. Yet, this could help if NaN loss somehow appeared a few times. A warning message would help to notify the user in such cases.

MFattouh avatar Jul 28 '18 10:07 MFattouh