net2net
net2net copied to clipboard
how to apply on new datasets
Hi,if I have a new dataset with source domain x and target domain y , how I train the model like creativity/portrait-to-photo
as your paper said, it should be train two autoencoder (Resnet101-as encoder, bigGAN as decoder)。
- use the source domain x data train an autoencoder and got encoder-x, decoder-x
- use the target domain y data train an autoencoder and got encoder-y, decoder-y
- use the pretrained model (encoder-x and decoder-y) train an cINN , it will learn translation z_encx to z_ency ?
is right ??
and would you provide an tutorial for how to apply on new datasets? thank you
Hi @Johnson-yue ,
Thanks for your message. The workflow you described is only applicable in the case where you have paired training data (as in the super-resolution example). Unpaired translation between domains (such as portrait-to-photo) is formulated as a translation between expert 1, a model which can infer the domain a given image belongs to, and expert 2, a model which can synthesize images of each domain. In the examples provided, we assume that the domain label comes with the dataset and provide the net2net.modules.labels.model.Labelator
module, which simply returns a one hot encoding of this label. However, one could also use a classification model which infers the domain label from the image itself. For expert 2, our examples use an autoencoder trained jointly on all domains, which is easily achieved by concatenating datasets together. The provided net2net.data.base.ConcatDatasetWithIndex
concatenates datasets and returns the corresponding dataset label for each example, which can then be used by the Labelator
class for the translation. We added the training configurations for the autoencoders used in the creativity experiments which can now be found in configs/autoencoder/anime_photography_256.yaml
, configs/autoencoder/celeba_celebahq_ffhq_256.yaml
and configs/autoencoder/portraits_photography_256.yaml
.
For unpaired translation on your own data, create pytorch datasets for each of your domains, create a concatenated dataset with ConcatDatasetWithIndex
(follow the example in net2net.data.faces.CCFQTrain
), train an autoencoder on the concatenated dataset (adjust the data
section in configs/autoencoder/celeba_celebahq_ffhq_256.yaml
) and finally train a net2net translation model between a Labelator
and your autoencoder (adjust the sections data
and first_stage_config
in configs/creativity/celeba_celebahq_ffhq_256.yaml
). You can then also add your new model to the available modes in the ml4cad.py
demo to visualize the results.
We've added this small tutorial also to the readme. Let us know if this helps or if you have further questions. Best Patrick