Zhao Tang comments

Repositories
Issues
Comments

Results 3 comments of


                                            Zhao Tang

Using accelerate launch to initialize sagemaker job doesn't work properly with multiple GPUs

If `accelerate launch` is invoked inside of sagemaker instead of used to create the sagemaker job, the script works fine. I suspect this is because MPI is not well-supported by...

Using accelerate launch to initialize sagemaker job doesn't work properly with multiple GPUs

> Yes, I'd recommend invoking inside of sagemaker instead in this case. (Though MPI should only be ran on CPU, not GPU) Sorry if I wasn't clear in my original...

Using accelerate launch to initialize sagemaker job doesn't work properly with multiple GPUs

@eyal-converge I presume you followed the instructions [here](https://huggingface.co/docs/accelerate/en/usage_guides/sagemaker) and used accelerate to **launch a Sagemaker training job**. The solution is instead of launching a job with accelerate, launch a job...