How to train on multiple GPU (w/ small vRAM)

[kebijuelun]

Thanks for the great work. Do you have any recommended solutions for running finetune.py code on multiple small memory GPUs (such like finetune lamma-30b on 8xV100)?

Is it possible for pipeline parallelism in deepspeed to be easily integrated into current code?

[zachNA2]

Have you tried using torchrun?

[kebijuelun]

Have you tried using torchrun?

I tried torchrun, but the DDP can not reduce the vRAM.

[8bit-coder]

I am interested in this as well, bumping for visibility

[taishiciR]

same question