Frederik Künstner

Hi Qiyao, Beware of `BatchNorm`; most of the quantities returned by BackPACK are not defined when there's a batchnorm layer in the middle (see e.g. https://github.com/f-dangel/backpack/issues/239). Easy checks of things...

> is there currently an alternative to BatchNorm? There are, for example GroupNorm or LayerNorm (see https://pytorch.org/docs/stable/nn.html#normalization-layers). The problem with BatchNorm is that there are no "individual gradient"; it is...

Hi Alex, thanks for the report! MSELoss not supporting vectors is annoying if it clashes with the standard pytorch API. I'll try to look into it. There's no technical reason...

Hi Thomas, I think the problem is that the error message is not strong enough. It should be ``` "Encountered BatchNorm module in training mode. Quantity to compute is undefined."...

Thanks for the check! I'd lean more towards crash that warn, but to get to something we can 👍; How about, starting from this setup; - Revert the default to...

Adding a `zero_grad` on the inputs gradients seem to fix the issue. As in changing the last few lines of the above script ``` optimizer.zero_grad() loss.backward() optimizer_alpha.step() ``` to ```...

You are right, the code only works for (linear+nonlinearity) layers. Treating other constructs such as RNNs or CNNs requires more work. The general concept still works: gather the information needed...

For [first-order extensions](https://docs.backpack.pt/en/master/extensions.html#first-order-extensions) (eg individual gradients) all loss functions or arbitrary transformations are supported (see [here](https://docs.backpack.pt/en/master/good-to-know.html#extend-ing-for-first-and-second-order-extension)). Transformations with parameters that need gradients (see [making a custom module](https://docs.backpack.pt/en/master/use_cases/example_custom_module.html)) or second-order quantities...

The docs have an example on [how to implement a custom module for first-order extensions](https://docs.backpack.pt/en/master/use_cases/example_custom_module.html). Second-order extensions have a bit more moving parts. Below is a script that implements the...

Not sure. Can you isolate the specific batch that is causing the issue? There shouldn't be a dependency across batch evaluations, so I'm not sure "running it multiple time" is...