Optimisers.jl icon indicating copy to clipboard operation
Optimisers.jl copied to clipboard

convenience constructors

Open CarloLucibello opened this issue 4 years ago • 2 comments

It could be convenient to have an optimizer constructor also performing initialization:

opt, optstate = Nesterov(params(m), lr=0.1)  

instead of

opt = Nesterov(0.1)
optstate = Optimisers.init(opt, params(m))  

CarloLucibello avatar Feb 04 '21 16:02 CarloLucibello

Seems like a good idea to me. If this is something we want, can I suggest kwargs again for the internal parameters like LR? Not necessary to achieve this, but if we have non-parameter positional arguments, I think the interface is cleaner with kwargs for parameters.

darsnack avatar Feb 08 '21 16:02 darsnack

With #30 this will read

optstate = Optimisers.setup(Nesterov(0.1), model)
for _ in 1:N
    grad = ...
    optstate, model = Optimisers.update!(optstate, model, grad)
end

and there is no further need to hold onto the optimiser struct alone.

Do we want anything else here? These are possible but perhaps confusing:

optstate = Nesterov(model, 0.1)

optstate = Nesterov(model; eta = 0.1)

Edit: also possible are:

optstate = Nesterov(0.1)(model)

optstate = Nesterov(rho=0.8)(model)

mcabbott avatar Jan 28 '22 05:01 mcabbott