Learning Hierarchical Semantic Image Manipulation through Structured Representations

This is official PyTorch implementation of NeurIPS 2018 paper Learning Hierarchical Semantic Image Manipulation through Structured Representations by Seunghoon Hong, Xinchen Yan, Thomas Huang, Honglak Lee.

Please follow the instructions to run the code.

Prerequisites

• Mac OS X or Linux
• NVIDIA GPU (make sure your GPU has 12G+ memory) + CUDA cuDNN

Installing Dependencies

• Install Pytorch
  • Note: This implementation has been tested with Pytorch 0.3.1.

  conda install pytorch torchvision cudatoolkit=9.0 -c pytorch
  

• Install TensorFlow
  • Note: This implementation has been tested with TensorFlow 1.5.

  pip install tensorflow-gpu==1.5
  

• Install Python Dominate Library

  pip install dominate
  

Data Preprocessing

• Please run the following script that creates two folders checkpoints/ and datasets/.

  bash setup.sh
  

• Please download the Cityscapes dataset from the official website (registration required). After downloading, please put these files under the datasets/cityscape/ folder and run the following script.

  python preprocess_city.py
  

• Please download the ADE20K dataset from the official website. After downloading, please put these files under the datasets/ade20k/ folder and run the following script.

  python preprocess_ade.py
  

Inference using a Pre-trained Box-to-Layout Generator

• You can download the pre-trained box-to-layout models, please run the following scripts.

  bash scripts/download_pretrained_box2mask_city.sh
  bash scripts/download_pretrained_box2mask_ade.sh
  

• Now, let us generate the manipulated layout from the pre-trained models. Please check the synthesized layouts under checkpoints/.

  bash scripts/test_pretrained_box2mask_city.sh
  bash scripts/test_pretrained_box2mask_ade.sh
  

Inference using a Pre-trained Layout-to-Image Generator

• You can download the pre-trained layout-to-image models, please run the following scripts.

  bash scripts/download_pretrained_mask2image_city.sh
  bash scripts/download_pretrained_mask2image_ade.sh
  

• Now, let us generate the manipulated image from the pre-trained models. Please check the synthesized images under checkpoints/.

  bash scripts/test_pretrained_mask2image_city.sh
  bash scripts/test_pretrained_mask2image_ade.sh
  

Joint Inference

• We provide a script to generate image using the predicted layout. Please check the synthesized images under results/ folder.

  bash scripts/test_joint_inference_city.sh
  

Training Box-to-Layout Generator

• If you want to train the box-to-layout generator on Cityscape dataset, please run the following script (usually it takes a few hours using one GPU).

  bash scripts/train_box2mask_city.sh
  

• If you want to train the box-to-layout generator on ADE20K dataset, please run the following script (usually it takes a few hours using one GPU).

  bash scripts/train_box2mask_ade.sh
  

Training Layout-to-Image Generator

• If you want to train the layout-to-image generator on Cityscape dataset, please run the following script (usually it takes one day using one GPU).

  bash scripts/train_mask2image_city.sh
  

• If you want to train the layout-to-image generator on ADE20K dataset, please run the following script (usually it takes one day using one GPU).

  bash scripts/train_mask2image_ade.sh
  

Issue Tracker

• If you have any question regarding our pytorch implementation, please feel free to submit an issue here. We will try to address your question as soon as possible.

Citation

If you find this useful, please cite our work as follows:

@inproceedings{hong2018learning,
  title={Learning hierarchical semantic image manipulation through structured representations},
  author={Hong, Seunghoon and Yan, Xinchen and Huang, Thomas E and Lee, Honglak},
  booktitle={Advances in Neural Information Processing Systems},
  pages={2713--2723},
  year={2018}
}

Acknowledgements

We would like to thank the amazing developers and the open-sourcing community. Our implementation has especially been benefited from the following excellent repositories:

• Pytorch CycleGAN and Pix2Pix: https://github.com/junyanz/pytorch-CycleGAN-and-pix2pix
• Pytorch Pix2PixHD: https://github.com/NVIDIA/pix2pixHD
• Torch ContextEncoder: https://github.com/pathak22/context-encoder