Variance reduction for (Renyi)ELBO

[OlaRonning]

From the forum thread. Tighter Variational Bounds are Not Necessarily Better demonstrates that tighter VBs can lead to deteriorating gradient estimates for VAEs trained with IWAE due to the variance decreasing slower than the expected value. DReG is a variance reduction technique that can alleviate this issue.

@fritzo and @eb8680 have a DReG draft for IWAE (#2605), from feature request #2598. Is it a good place to continue from?

@martinjankowiak you mentioned there may be other ways to revamp the ELBOs, did you have something particular in mind?

[martinjankowiak]

@OlaRonning well i believe the current renyi implementation is suboptimal (or incorrect, depending on your point of view) for a local latent variable model like a VAE. basically it has to do with order of summation/logarithm. if there are plates that enclose the entire model then the corresponding summations should be moved out but as it is i believe the implementation effectively assumes that there is a single global latent variable and doesn't make effective use of conditional independence

not sure if you should start from 2605 but you should definitely look at it carefully :)

[fehiepsi]

doesn't make effective use of conditional independence

This is right.