Selvaraj Anandaraj
Selvaraj Anandaraj
Can we add conditional checks instead of removing? We can do a torch NCCL version check.
Ack on your point about executor script launch. I also agree that's a problem. Why do we want to remove this? I don't want someone in the future to run...
>We recently had an internal team run into issues with Nemotron4 and checkpointing when those flags were set when using a Nemo container with NCCL 2.27.x+ Shouldn't this be an...
The goal of this kernel was to avoid saving the input for backward. The goal is to write the gradients on the input tensor itself to reduce the peak memory...