Multi GPUs training is very slow

[zscwind]

I used 4 GPUs on 1 node: torchrun --standalone --proc_per_node=4 train.py --compile=False But, the training speed is just like 1 GPU，why?

[han508]

Hello, have you solved it? I have the same problem as you

[sertannavdann]

Check => https://pytorch.org/docs/stable/amp.html