Detectron.pytorch
Detectron.pytorch copied to clipboard
Handle NaN loss
To avoid division by zero or passing zeros or negative values to log in bbox_transform_inv I replace those values with cfg.EPS. In addition, the NaN mask loss which rises after a negative area calculation is handled by discarding the entire batch.
This might be a tempting fix, but in my opinion if you have a NaN loss somewhere, then there is a serious problem in your network which deserves a deeper investigation.
There is probably a good reason why FAIR did not implement it that way in the first place.
well, you're absolutely right. NaN should not "normally" appear. Otherwise, A deeper investigation is required. Yet, this could help if NaN loss somehow appeared a few times. A warning message would help to notify the user in such cases.