About synchronize between difference gpus

[yifanjiang19]

Hi, I noticed that you use torch.nn.DistributedDataParallel. This program will launch n process (n=gpu number) and calculate the loss in different gpus individually. Is the synchronization (like torch.distributed.all_reduce()) needed before loss.backward() to synchronize the loss in all gpus?

Thanks!