MadNLP.jl LBFGS approximation of the inverse Hessian

Hi, thanks for this fantastic package. I have a question. I couldn't see any way to not pass the Hessian in and just rely on an LBFGS approximation of the (inverse) Hessian. Is there a way to do this in MadNLP now? If not, is it too much work to add?

Jun 16 '21 11:06 mohdibntarek

I am running into a similar issue. I think we could build something directly in the Hessian callback by wrapping the LBFGS operator implemented in JuliaSmoothOptimizers: https://github.com/JuliaSmoothOptimizers/LinearOperators.jl/blob/master/src/lbfgs.jl It's not direct to implement, but I think it's doable. In my opinion, the main difficulty is passing the LBFGS matrix to the KKT matrix before solving the linear system.

Jun 16 '21 11:06 frapac

I think we should use the Schur complement to solve the system. Assume D is the Hessian below.

The linear system M \ b can be solved easily if we have an operator x -> D^-1 * x.

Jun 16 '21 11:06 mohdibntarek

Thanks for bringing this up @mohamed82008 @frapac. I agree that MadNLP should have a quasi-Newton option. The linear operator implementation for LBFGS in JSO looks great; maybe we can use this.

It seems to me that this may take a good amount of time though. I'll try to implement this over the summer but can't promise anything yet. I'll keep this issue open and update the progress here.

Jun 18 '21 16:06 sshin23

LBFGS has been implemented in this PR: #221 At the end, we do not depend on JSO for the LBFGS implementation, and use instead the compact representation introduced in:

Byrd, Richard H., Jorge Nocedal, and Robert B. Schnabel. "Representations of quasi-Newton matrices and their use in limited memory methods." Mathematical Programming 63, no. 1 (1994): 129-156.

This is similar (with slight differences) to what is currently implemented in Ipopt.

Oct 05 '22 22:10 frapac

Solved by #221

Mar 27 '23 19:03 frapac

Been admiring this work from a distance. Glad it finally made it in 🎉

Mar 28 '23 10:03 mohdibntarek

Thank you for your support! We would love seeing MadNLP integrated in NonConvex.jl in the medium term :)

Mar 28 '23 11:03 frapac

Maybe I'm missing something but, should it means that user-defined function with JuMP + MadNLP should work now ?

I'm still getting the Hessian information is needed. error using JuMP 1.11.1 and MadNLP 0.7.0

Related to : #115

edit : I'm not sure that you still receive notification after the issue is closed so : @frapac @sshin23

Thanks

Jun 13 '23 17:06 francis-gagnon

@francis-gagnon indeed, I haven't received your comment before. MadNLP is currently not supporting user-defined operators inside MOI. This is resolved with this new PR: https://github.com/MadNLP/MadNLP.jl/pull/322

Apr 10 '24 20:04 frapac