Michael Abbott

Perhaps too snarky, sorry; the higher order case was before I paid this package any attention. If that `AdaHessian` works then someone should at least add it to the tests....

It used to use `eps(typeof(eta))`, which had the unfortunate effect that `Adam(1e-4)` and `Adam(1f-4)` did different things. Even if your model was all Float32, you could easily provide `eta::Float64` by...

With #30 this will read ```julia optstate = Optimisers.setup(Nesterov(0.1), model) for _ in 1:N grad = ... optstate, model = Optimisers.update!(optstate, model, grad) end ``` and there is no further...

Haven't had a chance to look. My guess is that somehow the mutation of the BatchNorm struct is lost by Optimisers. Would be interesting to know if https://github.com/FluxML/Flux.jl/pull/2127 changes this,...

> consistently with rules.jl I don't think that's right. At present, `state` is what's produced by `init` and stored in one `Leaf` in a field of that name. Whereas `tree`...

> Flux usage example in this package's documentation This may well pre-date the re-write -- at some point everything was called `state`, and there was no Leaf, there was less...

> Inserting this struct in place of leaf nodes instead To be clear I've only considered reversible modifications. So the closest thing is 1.,https://github.com/FluxML/Optimisers.jl/pull/49 which replaces the Leaf with a...

The `optax.masked` is interesting. If I understand right, this is closer to a modification of our `setup`, to apply different rules to different parameters. (Or to append more rules to...

Values from the cache are used when an object `x` is `===` some previously seen `x`. They should therefore always have the same type as what `init(rule, x)` returns. If...

```julia function _setup(rule, x; cache) if haskey(cache, x) T1 = Base._return_type(init, Tuple{typeof(rule), typeof(x)}) T2 = Base._return_type(Leaf, Tuple{typeof(rule), T1}) return cache[x]::T2 end if isnumeric(x) ℓ = Leaf(rule, init(rule, x)) # as...