Optimization.jl
Optimization.jl copied to clipboard
Include `searchdirection` in `OptimizationState`
It would be convenient if OptimizationState included a searchdirection
and/or a step width. I can usually pull these quantities out of the original
state, but it would be nice to have them in a well-defined location.
For simple gradient descent, the search direction would be the negative gradient, but e.g. for LBFGS it would be the direction obtained from the internal quasi-Hessian. Knowing the search direction allows to really drill down into the linesearch, see, e.g., the plots in the README of GRAPELinesearchAnalysis.jl
. In principle, I can infer the search direction as the difference in u
from the previous iteration, but then I don't know the step width (how far the optimizer went in the direction of the search direction). Knowing both quantities directly from the solver is useful for actually checking that the update is in the search direction, but if that's guaranteed, only one of them would be sufficient.
I can usually pull these quantities out of the original state
Well, I thought I could, but it turns out the state
is woefully inadequate, at least for some optimizers. I'm dumping state
in the callback in an optimization with NLopt.LD_LBFGS()
and seeing this:
Optimization.OptimizationState{Vector{Float64}, Float64, Nothing, Nothing, Nothing}
iter: Int64 0
u: Array{Float64}((9,)) [0.13823471888875286, 0.30634894494361153, 0.1562189341456497, 2.6013349121108944, 2.601329761795745, 3.6818511117018, 4.997713711972462, 4.989897143702104, 4.9818589345359605]
objective: Float64 0.01069630598006488
grad: Nothing nothing
hess: Nothing nothing
original: Nothing nothing
So it's missing a lot of information, and doesn't forward any internal LD_LBFGS
state.
Yes, we need to fill it in a bit more per optimizer, and not all optimizers have al linformation.