Add API for (log) marginal likelihood approximation

[st--]

AbstractGPs currently defines elbo, but other approximations (e.g. Laplace, EP) also provide approximations to the (log) marginal likelihood that can be used for hyperparameter optimisation. Would be great to have a unified API function for that (for VFE, SVGP, etc it could simply forward to elbo). What do you think? Finding a decent name seems the most complicated aspect of this:

• approx_lml (short but a bit cryptic)?
• approx_log_marginal_likelihood (explicit but so long)?
• approx_log_marglik (the compromise that makes everyone unhappy)?
• approx_log_evidence?

[theogf]

I like the idea! In AGP I simply have objective but that's too vague. I like approx_log_evidence, but what about surrogate instead of approx? "Approximate" is a bit inaccurate in my opinion since it does not indicate that we are working with a bound...

[st--]

"Approximate" is a bit inaccurate in my opinion since it does not indicate that we are working with a bound...

My point was that in some cases it really is just an approximation (without guarantees on how good it is), it's not necessarily a bound (and if it's a lower bound indeed, I guess elbo is fine/sufficiently commonly known?).

[st--]

Should we define approx_log_evidence(::FiniteGP, ...) = logpdf(...)?

[theogf]

it's not necessarily a bound

With which method it is not a bound?

[willtebbutt]

With which method it is not a bound?

Pretty much anything that's not the ELBO. EP (FITC) / SoR / DTC etc are all just approximations, with no particular guarantee as to whether they're greater or less than the log marginal prob.

[st--]

I'm just thinking, how about moving VFE & DTC out of AbstractGPs and into ApproximateGPs?

[willtebbutt]

I was wondering about that too. On the face of it, it does make more sense, as they're certainly approximations. The only downside I can see is that you'll have the dependencies associated with ApproximateGPs as well as those for AbstractGPs if you want to extend the VFE for your own AbstractGP subtype (as I do in TemporalGPs)

[st--]

Should we define approx_log_evidence(::FiniteGP, ...) = logpdf(...)?

This doesn't quite make sense to me anymore, as it would require a different signature from the approximate GPs. Instead, we could have an ExactGP "approximation" type so that approx_log_evidence(ExactGP(), fx::FiniteGP, y::AbstractVector{<:Real}) = logpdf(fx, y), and posterior(ExactGP(), fx, y) = posterior(fx, y), if people want to compare e.g. ExactGP vs VFE/DTC/...