ShivamShrirao
Running training on multiple GPUs
Currently, there does not seem to be support for a parallelized training on multiple GPUs. As far as I know, the from_pretrained loading of models and pipelines should allow us to use device_map=auto, which neatly distributes the computing resources onto multiple GPUs.
I've tried to (naively) insert the device_map=auto into all calls for from_pretrained with little success. As expected, I'm receiving a Expected all tensors to be on the same device, but found at least two devices, cuda:1 and cuda:0!. I know to little of what is going on with all the device calls to properly implement a --multiple_gpu=True argument, but I'd love to see that. If anyone attempts, you should know that CLIPTextModel does not support device_map=auto as of today.
Maybe there is even an easier way of parallelizing training?
