how to apply on new datasets

[Johnson-yue]

Hi，if I have a new dataset with source domain x and target domain y ， how I train the model like creativity/portrait-to-photo

as your paper said， it should be train two autoencoder （Resnet101-as encoder， bigGAN as decoder）。

1. use the source domain x data train an autoencoder and got encoder-x， decoder-x
2. use the target domain y data train an autoencoder and got encoder-y， decoder-y
3. use the pretrained model （encoder-x and decoder-y） train an cINN ， it will learn translation z_encx to z_ency ？

is right ？？

and would you provide an tutorial for how to apply on new datasets？ thank you

[pesser]

Hi @Johnson-yue , Thanks for your message. The workflow you described is only applicable in the case where you have paired training data (as in the super-resolution example). Unpaired translation between domains (such as portrait-to-photo) is formulated as a translation between expert 1, a model which can infer the domain a given image belongs to, and expert 2, a model which can synthesize images of each domain. In the examples provided, we assume that the domain label comes with the dataset and provide the net2net.modules.labels.model.Labelator module, which simply returns a one hot encoding of this label. However, one could also use a classification model which infers the domain label from the image itself. For expert 2, our examples use an autoencoder trained jointly on all domains, which is easily achieved by concatenating datasets together. The provided net2net.data.base.ConcatDatasetWithIndex concatenates datasets and returns the corresponding dataset label for each example, which can then be used by the Labelator class for the translation. We added the training configurations for the autoencoders used in the creativity experiments which can now be found in configs/autoencoder/anime_photography_256.yaml, configs/autoencoder/celeba_celebahq_ffhq_256.yaml and configs/autoencoder/portraits_photography_256.yaml.

For unpaired translation on your own data, create pytorch datasets for each of your domains, create a concatenated dataset with ConcatDatasetWithIndex (follow the example in net2net.data.faces.CCFQTrain), train an autoencoder on the concatenated dataset (adjust the data section in configs/autoencoder/celeba_celebahq_ffhq_256.yaml) and finally train a net2net translation model between a Labelator and your autoencoder (adjust the sections data and first_stage_config in configs/creativity/celeba_celebahq_ffhq_256.yaml). You can then also add your new model to the available modes in the ml4cad.py demo to visualize the results.

We've added this small tutorial also to the readme. Let us know if this helps or if you have further questions. Best Patrick