log marginal likelihood objective and posterior GP

[st--]

trafficstars

When preparing #240 I stumbled across having to pass both FiniteGP for input features f(x) and targets y to both posterior (for PosteriorGP) and logpdf (for log marginal likelihood). Maybe the answer is no, but should we have something like logpdf(::PosteriorGP)?

Also, is it sufficiently intuitive that you get the log marginal likelihood by calling logpdf? why would calling the probability density function return what we get when we integrate out the uncertainty over the GP?

[willtebbutt]

When preparing #240 I stumbled across having to pass both FiniteGP for input features f(x) and targets y to both posterior (for PosteriorGP) and logpdf (for log marginal likelihood). Maybe the answer is no, but should we have something like logpdf(::PosteriorGP)?

There's nothing to stop us doing this. It obviously can't be part of any of the AbstractGPs APIs, but that's not to say we shouldn't add a special case thing for PosteriorGP.

Also, is it sufficiently intuitive that you get the log marginal likelihood by calling logpdf? why would calling the probability density function return what we get when we integrate out the uncertainty over the GP?

logpdf(fx, y) says "give me the log probability density of y under the fx, where fx is the marginal distribution over f at x`, which has always felt clear to me (i.e. the log marginal probability density is just a log probability density under the marginal distribution).

Do you have a particular concern?

[willtebbutt]

I was thinking about this again in the context of approximate inference.

I think I would be in favour of adding a function called log_evidence(::PosteriorGP) and (something along the lines of ) approx_log_evidence(::ApproxPosteriorGP). This would give us the best of two worlds:

1. it would maintain compatibility with the Distributions interface, i.e. we'd retain logpdf(fx, y), which is needed for compatibility with lots of things, and
2. we would resolve the issue of needing to recompute intermediate quantities if you want both the log evidence and access to the posterior.

It's also quite clear what log_evidence ought to mean in the context of a PosteriorGP.

What do you think @st-- ?