sylviachency
sylviachency
> I tried using lora. Ran into OOM issue with the same machine configuration (8 A100 80GB GPUs) > > Just curious would the model even fit in a single...
There’re previous versions that support multi gpu? On Sat, Jul 1, 2023 at 01:13 weilong ***@***.***> wrote: > Just check the code, it's not implemented for multi-gpus. I guess he...
> Sorry, I was out for the last couple of days. Yes, previous versions tried to support multi GPUs (through FSDP, DeepSpeed) but none of them worked so it's been...
Hi. Lora distributed is not implemented. How did you train with lora with 2 A100?
Did you see memory occupation on both GPUs? If you look into it’s source code for Lora, there’s a “NotImplemented Error” when device > 1. So to my understanding in...