mlcolab
Implement the AdvancedVI interface
Even though Pathfinder is intended for initializing MCMC, it is a variational method, so we should implement the AdvancedVI.jl interface.
Currently AdvancedVI is missing an API function we would need (see https://github.com/TuringLang/AdvancedVI.jl/issues/32). Also, it seems to be in the middle of a rewrite (see https://github.com/TuringLang/AdvancedVI.jl/pull/25), so it might be worth waiting on this until the rewrite is finished.
While the AdvancedVI interface seems now complete, the main issue now is that AdvancedVI expects us to be able to have a step function, which would here correspond to a new optimization step followed by generation of a variational approximation and estimate of ELBO. The problem now is Optimization doesn't have any interface for iterative optimization; it's only solve API is through solve/solve!, which loops over all iterations until termination. Examples are https://github.com/SciML/Optimization.jl/blob/7a286a4392313dd9192f4e55882986fafa66808d/lib/OptimizationOptimJL/src/OptimizationOptimJL.jl#L209 and https://github.com/SciML/Optimization.jl/blob/7a286a4392313dd9192f4e55882986fafa66808d/lib/OptimizationLBFGSB/src/OptimizationLBFGSB.jl#L165
It's not quite implementing the full AdvancedVI API, but we could just overload {AdvancedVI.optimize](https://turinglang.org/AdvancedVI.jl/stable/general/#AdvancedVI.optimize) in an extension instead of using AdvancedVI's own optimization implementation. However, I don't really see what the benefit would be.