Probabilistic PCA is not identifiable

[dcmoyer]

Hi there, thanks for the great book!

In section 28.3.1.6, there's a discussion of Factor Analysis, so this two Gaussian model: $p(z) = \mathcal{N} (0,1)$, $p(x|z) = \mathcal{N} (x | Wz + \mu, \Psi)$.

In the section, the book states correctly that this model is not identifiable because for a new $\tilde{W} = WR$ for any rotation matrix R, we have the same likelihood (since $z$ is isotropic). However, it then goes on to say that probabilistic PCA becomes identifiable with an orthogonality constraint on $W$. However, this isn't true, since $\tilde{W} = WR$ for rotation $R$ and orthogonal $W$ is also orthogonal, so for every $W$ there are multiple $\tilde{W}$ with equal likelihood, even under the orthogonality constraint.

Lecture notes that agree, saying PPCA is not identifiable: [pdf] https://www.cs.princeton.edu/~bee/courses/lec/lec_feb21.pdf (note that here F is only defined to be normal after the fact...maybe an error during lecture or by the scribe) [pdf] https://www.cse.iitk.ac.in/users/piyush/courses/pml_winter16/slides_lec10.pdf