Optimisers.jl
Optimisers.jl copied to clipboard
convenience constructors
It could be convenient to have an optimizer constructor also performing initialization:
opt, optstate = Nesterov(params(m), lr=0.1)
instead of
opt = Nesterov(0.1)
optstate = Optimisers.init(opt, params(m))
Seems like a good idea to me. If this is something we want, can I suggest kwargs again for the internal parameters like LR? Not necessary to achieve this, but if we have non-parameter positional arguments, I think the interface is cleaner with kwargs for parameters.
With #30 this will read
optstate = Optimisers.setup(Nesterov(0.1), model)
for _ in 1:N
grad = ...
optstate, model = Optimisers.update!(optstate, model, grad)
end
and there is no further need to hold onto the optimiser struct alone.
Do we want anything else here? These are possible but perhaps confusing:
optstate = Nesterov(model, 0.1)
optstate = Nesterov(model; eta = 0.1)
Edit: also possible are:
optstate = Nesterov(0.1)(model)
optstate = Nesterov(rho=0.8)(model)