double7 comments

Repositories
Issues
Comments

Results 2 comments of


                                            double7

[bugfix] Force create checkpoint directory before saving dataloader state.

https://github.com/volcengine/verl/blob/bb4f97b7541c52936844a4522d6bb6d8570c2c10/verl/trainer/ppo/ray_trainer.py#L846-L856 I checked the code to determine why this error occurred. In my run, L846 executes successfully, which happens in each worker and the directory is automatically created by `FSDPCheckpointManager`....

Race condition when creating checkpoint directories causes training failures

Meet the same problem