AlphaBetaGamma96

@fuzihaofzh Is the hessian with respect to the inputs of your model or the parameters of your model? There is a trick for the inputs but none exists (to my...

Hi @fuzihaofzh, I came across this repo (https://github.com/amirgholami/adahessian) and thought you might find it interesting as it uses a trick for the Hessian via Hutchinson's estimator.

I'm working on some gradient preconditioning techniques and it requires the forward activations and backward sensitivities (grad_output) of all nn.Module objects of a network. I am also calculating per-sample gradients...

@zou3519 Do you know if it's possible to compute the `grad_output` of a layer via `vmap` or is it only possible via hooks?

Hi @albanD, thanks for the insight on how to use `register_hook`! Just so I understand this correctly, I can register a hook on a Tensor and use my `forward_pre_hook` and...

Hi @albanD, I had a quick look at `register_hook` but it seems that the signature of that hook only takes the gradient of a Tensor rather than the `grad_output` values...

So, what I need is the gradient of the output of a module (which is what full_backward returns, although doesn't work with functorch atm) but after checking `register_hook` it seems...

That's a neat trick! So this would basically do what `full_backward_hook` does but for an `nn.Module` object with only 1 output and works with functorch? That kinda looks like what...

No need to apologize for the delay @samdow, thanks for solving the batch rule! Fingers crossed it all works!

Hi @samdow, thanks for fixing this issue! A bit of a silly question, but I remember reading somewhere that functorch is being merged directly into pytorch (if that's the correct...