Question about marginal_likelihood in VAE

[dongfeng951]

Thank you for your working and sharing. I learned a lot from them. However, I have a question about the VAE implementation. In VAE.py, you calculate the marginal_likelihood as an cross entropy: marginal_likelihood = tf.reduce_sum(self.inputs * tf.log(self.out) + (1 - self.inputs) * tf.log(1 - self.out),[1, 2]) However, I am confusing as the formular is : where the first item on the right side should correspond to the marginal_likelihood. I think the latent variable z should be included to calculate the likelihood but you use the initial input, self.inputs. So I am a little confusing, can you explain it? Thank you very much!

[hwalsuklee]

Hi.

There are two tricks on implementation of marignal likelihood. 1st is to use Monte-carlo technique instead of integral. 2nd is to use only one sample for Monte-carlo technique.

With these tricks, marginal likelihood is approximated to log(p(x_i|z)). Since p(x_i|z) is assumed to follow bernoulli distribution, log(p(x_i|z)) is cross entropy between network input and output.

Please check following.

[dongfeng951]

@hwalsuklee Hi, thanks for the explanation. I understand it. I have another question. There is a KL gap between the ELBO and p(x), so how can I compute the p(x)? Thank you!

[vedal]

Dear @hwalsuklee: do you know how negative log-likelihood for f.ex Cifar test set is reported in VAE-like models? Paper examples etc are included in my question here: https://www.reddit.com/r/MLQuestions/comments/9sp6d3/how_to_calculate_log_likelihood_for_vaevqvae/