HRNet-Object-Detection icon indicating copy to clipboard operation
HRNet-Object-Detection copied to clipboard

Published 20 hours ago • verified publisher icon HRNet

training suddenly get slower and slower

Open Wangzhuoying0716 opened this issue 5 years ago • 0 comments

Hi, I have met a problem when I tried to train on 2 RTX 8000 GPU card. The first epoch is normal and it prints the log about every 1 minutes. But when it comes to the 2 or 3 epoch, the time becomes 4~5 minutes and the extra presented training time is getting longer. I killed it and then train with 'resume'. It's normal again but after 1 epoch and some iterations, it again gets slower and slower. I'm confused and grateful to get some help !

Wangzhuoying0716 avatar Nov 25 '19 13:11 Wangzhuoying0716