matthewdeng

Results 23 comments of matthewdeng

[Train] TorchTrainer does not free all GPUs on shutdown

The team is looking into properly terminating subprocesses, but more investigation is needed to understand how to do so. Though based on your original observations and the discussion in the...

ray.tune hangs instead of quitting when Attempting to recover lost objects by resubmitting their tasks, not releasing the main process

@iycheng do you have any ideas about the Core failure?

[RayTrain] Checkpoint API to recover from checkpoint from previous runs

@sathyanarays is [this](https://docs.ray.io/en/latest/train/user-guides/fault-tolerance.html#auto-resume) the API you'd be looking for?