Understanding and Improving Interpolation in Autoencoders via an Adversarial Regularizer

[evancofer]

Autoencoders provide a powerful framework for learning compressed representations by encoding all of the information needed to reconstruct a data point in a latent code. In some cases, autoencoders can “interpolate”: By decoding the convex combination of the latent codes for two datapoints, the autoencoder can produce an output which semantically mixes characteristics from the datapoints. In this paper, we propose a regularization procedure which encourages interpolated outputs to appear more realistic by fooling a critic network which has been trained to recover the mixing coefficient from interpolated data. We then develop a simple benchmark task where we can quantitatively measure the extent to which various autoencoders can interpolate and show that our regularizer dramatically improves interpolation in this setting. We also demonstrate empirically that our regularizer produces latent codes which are more effective on downstream tasks, suggesting a possible link between interpolation abilities and learning useful representations.

https://arxiv.org/abs/1807.07543v2

[evancofer]

This seems highly relevant to the section on latent space manipulation.

[stephenra]

Thanks @evancofer for bringing this up. In the broader context of interpretable semantic information as well as traversing the latent space representation, would it be informative to add papers addressing disentanglement?

Tagging @gwaygenomics given his experience and research in this area.

[gwaybio]

thanks @stephenra - definitely near the top of the list! Once I read, I will add a summary here

[gwaybio]

Ok, I put aside some time this afternoon to give this a quick read. The article was very well written. I will add my summary below:

Computational Methods

• The authors introduce a novel autoencoder loss function they call "Adversarially Constrained Autoencoder Interpolation" or ACAI.
  • The autoencoder loss function adds a term that tries to maximize tricking a critic (discriminator) model into predicting that an interpolated point is real data
• The authors introduce an evaluation procedure using image data with a known underlying manifold
  • the images consist of a black background and a single white line that can rotate 360 degrees
  • the authors apply several different autoencoder architectures to this task and evaluate performance
    • In general, they provide an excellent (and succinct) description of many autoencoder flavors - I think serves a a great reference for autoencoder newcomers
• A couple of notes on this specific evaluation:
  • Denoising autoencoders don't actually interpolate well - they approximate the "data space" interpolation (interpolation in the input raw data space instead of the latent space)
  • Variational autoencoders generate realistic interpolations, but they are not actually "smooth" (the interpolations do not sequentially rotate)
  • In a separate eval, the VAEs and DAEs generate good representations that can be input into supervised learning task with good predictive performance

Biological Relevance

There are many biological and biomedical applications in which accurate interpolation between existing data points can provide immediate benefit. Similar benchmarking of models in interpolation tasks in biological domains is definitely of interest. In the present paper, there are no references to specific biomedical applications, but it may be worth mentioning in the latent space manipulation section as directly applicable future directions.