nanoGPT icon indicating copy to clipboard operation
nanoGPT copied to clipboard

Published 20 hours ago • verified publisher icon karpathy

Multi GPUs training is very slow

Open zscwind opened this issue 1 year ago • 2 comments

I used 4 GPUs on 1 node: torchrun --standalone --proc_per_node=4 train.py --compile=False But, the training speed is just like 1 GPU，why?

zscwind avatar Feb 08 '23 14:02 zscwind