Ranger21 icon indicating copy to clipboard operation
Ranger21 copied to clipboard

Error when using DDP

Open smartbarbarian opened this issue 1 year ago • 1 comments

I used ZeroRedundancyOptimizer to wrap Ranger21 and running it in a 4-GPU machine. But its performace is much worse compared with the simple AdamW and it showed the error. loerror in warmdown pct calc. new pct = 67.11272727272727���������������������������������������������������������������� | 19156/19505 [8:57:59<04:55, 1.18it/auto handled but please report issue error in warmdown - lr below min lr. current lr = 2.999999999999997e-05 auto handling but please report issue! error in warmdown pct calc. new pct = 67.11272727272727 auto handled but please report issue error in warmdown - lr below min lr. current lr = 2.999999999999997e-05 auto handling but please report issue! error in warmdown pct calc. new pct = 67.11272727272727 auto handled but please report issue error in warmdown - lr below min lr. current lr = 2.999999999999997e-05 auto handling but please report issue! error in warmdown pct calc. new pct = 67.11272727272727 auto handled but please report issue error in warmdown - lr below min lr. current lr = 2.999999999999997e-05 auto handling but please report issue!

smartbarbarian avatar May 08 '23 02:05 smartbarbarian

Did you find a solution to this? I have the same problem.

TolerantChief avatar Jun 11 '23 16:06 TolerantChief