Xuefeng Li

https://discuss.pytorch.org/t/extra-10gb-memory-on-gpu-0-in-ddp-tutorial/118113 I put these two lines of code at the beginning of the training_loop function, and the problem is solved. torch.cuda.set_device(rank) torch.cuda.empty_cache() Hope this will help you.