FewX icon indicating copy to clipboard operation
FewX copied to clipboard

Published 20 hours ago • verified publisher icon fanq15

NaN

Open Retiina opened this issue 4 years ago • 1 comments

Dear author, thanks for you great work. Currently I am trying to run your code but always report NaN error, the following is the error traceback, can you have a look? Thanks in advance!

detectron2/detectron2/modeling/proposal_generator/proposal_utils.py", line 103, in find_top_rpn_
proposals
    raise FloatingPointError(
FloatingPointError: Predicted boxes or scores contain Inf/NaN. Training has diverged.

Retiina avatar Jan 19 '21 02:01 Retiina

When I train with 4 GPU, NaN gone. However, when I train on 8GPU, it always yield NaN

Retiina avatar Jan 19 '21 08:01 Retiina