pranathi chunduru comments

Repositories
Issues
Comments

Results 2 comments of


                                            pranathi chunduru

Multi GPU timeout on save checkpoint (WorkNCCL, Watchdog, timeout)

I am still seeing the issue with distributed lora on Llama-3.1-70B. I tried to follow https://github.com/pytorch/torchtune/issues/2093#issuecomment-2509733176 link to increase the timeout. I do have sufficient CPU-RAM to save the model-checkpoints....

Multi GPU timeout on save checkpoint (WorkNCCL, Watchdog, timeout)

> [@pranathichunduru](https://github.com/pranathichunduru) , can you please share a longer log? This is from the start of the process: ``` Setting manual seed to local seed 2241663802. Local seed is seed...