imraviagrawal

Results 4 comments of imraviagrawal

Save model checkpoint error when multi-gpu training still happens on 4.36.1

I see the error on 4.36.2 version as well, and I have a shared file system across each node. Using 2 nodes with 8 H100 gpus on each nodes.

Get stucked on multi GPUs training

Did you solve it, I am having same issue

when using huggingface pretrained model with multi-gpu, model parameters were duplicate for every gpu in ram

> Hey @linyubupa > > This is expected in the way you are initializing the model. I can see from the code snippet that you create the model in `__init__`....

Step3: 8 * A100-40G training LLAMA2-7B and OPT-350M out of memory.

Are you able to solve this error??