NVIDIA
Is there a way to get all requires_grad=True variables and do variable or gradient clipping inside simulator of warp？
Hi Is there a way to get all requires_grad=True variables and do variable or gradient clipping inside simulator of warp？Thank you!
Hmm.. that sounds useful, are you using the wp.Tape()? That keeps track of all the gradients used during the computation. You could add a post process to clip the gradients there I think?
Hi Miles Thank you for the reply. I will look at it. Btw, when I played warp and pytorch, I found warp cannot support pytorch's retain_graph=True functionality, i.e., loss.backward(retain_graph=True). But not using retain_graph solves the issue.
Hmm.. that sounds useful, are you using the
wp.Tape()? That keeps track of all the gradients used during the computation. You could add a post process to clip the gradients there I think?
When I try to change the value of gradients inside Tape function, it showed that 'operation would make the legacy stream depend on a capturing blocking stream'. Seems like I cannot do any modification of gradients' value.
