Carlos Mocholí
Carlos Mocholí
Please, let's keep the torchsnapshot integration focused to #14503. It's in our roadmap, just waiting for Lite changes to be over.
Can you share the complete error stacktrace?
Can you get a cleaned-up stacktrace? The one you shared is a bit unreadable. A screenshot would be fine too.
It is merged now :tada:
Interested in this issue! Hopefully some progress is done soon :+1:
@JiesiZhao077 Running in one GPU would be the easiest and safest option. You can also set this `DistributedSampler` which would also solve the issue, but introduces the risk of deadlocks....
To recap, the plan would be: - Enable "join" as an optional feature of the DDP strategy: `Trainer(strategy=DDPStrategy(uneven_input_support: bool)`. We could also add a registry string for it. - Add...
@jgbos You chose to get rid of the launcher subclass, right? > We could then make a separate strategy hydra_ddp. Why would we want this? This should be invisible to...
The `subprocess_cmd` refactor makes sense to me. But I would definitely avoid the need to introduce a separate strategy. The fact that Hydra needs a custom launch command does not...
Finally merged! Thank you so much for your time @jgbos