vqgan mode initializes image with washed-out brown palette

[dmarx]

suspect this is tied related to the black and white palette observed with init images in limited palette mode. i.e. there's probably some broader issue with how image state is initialized from init images

[dmarx]

washed-outbrown could be random latent? if that's the case, the issue is that if we have an init image, we should initialize from the latent from inverting the image. I thought that's what we were already doing, but need to make sure I guess.

[dmarx]

early frames are in fact the GAN-inverted init image. maybe it would help if we EMA'd the latent?

[dmarx]

tried cranking up the EMA, helped a tiny bit maybe. looks like the real help comes from turning up the direct image weight. I think what' going on here is that the direct image weight is being overpowered by one of the other weights like TV or MSE or whatever. or maybe the learning rate should be higher?...that can't be right.

[dmarx]

maybe this is the MSE regulation kicking in? if so, maybe we could get around this by adopting the "smart_encode" strategy used in PixelImage.encode_image?