implement GP-LVM from scratch

[murphyk]

Currently fig 28.19 is created by https://github.com/probml/pyprobml/blob/master/notebooks/book2/28/gplvm_mocap.ipynb which uses GPy from Sheffield. Here is the core code

model = GPy.models.GPLVM(Y, 2, init="PCA", normalizer=True)
model.optimize(optimizer="lbfgs", messages=True, max_f_eval=1e4, max_iters=1e4)

It would be useful to reimplement this from first principles, using eg TinyGP to define the Gram matrix.

[murphyk]

BTW I slightly rewrote the section n GPLVM (28.3.7) to make the notation clearer. New version below.

I found this colab (based on PyMc4 , which uses TFP and is deprecated) that fits the GP-LVM model using ADVI to the iris dataset. https://colab.research.google.com/drive/1BhqYin-3W1bmGgNxA95HKZZw3ACmicRK?usp=sharing (They use a NormalCholesky prior on the latent inputs Z/X.) When using a linear kernel you get the same results as PCA (modulo rotation). This would be a good example to try to reproduce. (It seems from this thread that the code was never merged, presumably because pymc4 is deprecated.)