Is multi-gpu training supported?

[chiwing4]

I have 3 GPUs and when I run train.py, I can see all GPUs is working (Load is increased). However, I do not see any speed gain. Is there a correct way to do multi-GPU training?

Also, I found that train_loader.batch_sampler.set_epoch(epoch) is commented out inside train.py, and DistributedSampler is not used so I'm not sure if DDP is working properly.

Thanks