daib13
Does reconstruction loss dominate in the 2nd stage VAE?
Hello, I recently read this paper and found it fascinating and very relevant to my work. I am a little bit confused by one aspect of your approach, however. I understand the intuition that in a standard VAE, the reconstruction term dominates such that the model learns a useful latent representation of encoded data but fails to structure this space in a way that allows sampling novel, high-quality data.
My concern is that, what prevents the second stage VAE from falling into this same trap? Isn't it possible that the reconstruction loss on z will dominate in the second stage, so that it can encode and decode samples from the stage one posterior, but fail to structure the second latent space q(u) as normally distributed so that we can't generate samples of z from the "empirical prior"?
