Zach Mueller comments

Results 368 comments of


                                            Zach Mueller

Significant performance degradation with multi-GPU training on newer torch/transformers

Let me know if that does anything please 🙏

Significant performance degradation with multi-GPU training on newer torch/transformers

Oh boy, okay. Well, that was a thought 😢 CC @stas00 if I misunderstood anything? (See the nccl issue)

Trainer using only one GPU instead of two

How are you launching the python script in your bash?

Trainer using only one GPU instead of two

You're launching with python. You should use either `accelerate launch` or `torch.distributed.run` otherwise you'll get model parallel (which isn't what you're aiming for)

remove unnecessary backend related checks in training_args.py

Accelerate should handle most of this now, cc @SunMarc if you want to give this a try!

remove unnecessary backend related checks in training_args.py

@janboeye yes PyTorch does not have mixed precision support on MPS at this time

Extend save_pretrained to offloaded models

Overall I can see this being a pretty nice idea. Made some nits to improve. cc @SunMarc for your thoughts as well :)

[Bug] Error when trying to run two models in a machine with created ZeRO config variables.

At this time we do not support multiple models with `deepspeed`, please see: https://github.com/huggingface/accelerate/issues/2496