TransformerEngine icon indicating copy to clipboard operation
TransformerEngine copied to clipboard

Published 20 hours ago • verified publisher icon NVIDIA

Why requires_grad attribute of weight from offloading will set to False ?

Open Sakura-gh opened this issue 7 months ago • 1 comments

https://github.com/NVIDIA/TransformerEngine/blob/e3bb24e5a347c58353e62307bc84cca856f9e9be/transformer_engine/pytorch/module/linear.py#L405-L407

if the weight.requires_grad set to False, when to calculate and accumulate weight grads?

Sakura-gh avatar Jul 08 '24 08:07 Sakura-gh