pytorch-CycleGAN-and-pix2pix

pytorch-CycleGAN-and-pix2pix copied to clipboard

Hello all,

first of all as big thank you to people like @junyanz for publishing and maintaining this repo.

I am running some experiments with CycleGAN aiming to generate IR/thermal images from RGB. I have noticed that although the reconstructed RGB image is of high quality, yet the generated IR image is quite poor in terms of domain characteristics. Actually it seems like the generated IR image is more like a grayscaled version of RGB input, where the object shapes are quite clear but the pixel values do not correspond to the domain identities, e.g. warm objects like people or moving cars should have high pixel values but in many cases they are translated much darker based on the original RGB color.

I suppose this is an expected issue since CycleGAN process the data during training in an unpaired manner. Furthermore, the training loss is mostly based on the quality of the reconstructed RGB image, so the model can learn an intermediate mapping to IR domain, which might not be realistic but it can be convenient enough to reconstruct RGB image from it.

I would like to ask the community and especially @junyanz if this explanation sounds reasonable and if there are any suggestions to improve the generated images quality.

Cheers!

Sorry for the late reply. It's a little hard to know what's going on without seeing your images and results. You may want to have a close look at the GAN loss plot. Also consider (1) lowering cycle-consistency loss, (2) using a small number of paired data, (3) using additional auxiliary information (semantic label map)

@wave-transmitter @junyanz I also used IR to convert RGB and found rec_ B works well, but fake_ B is the same as the IR input, and the color becomes very dark. How to solve this? Similarly, I use the cyclegan model. My data resolution is 2560 * 1440. 160 training data and 40 test data. Here are my training and testing commands:

1. python train.py --dataroot ./single/testIR --name test_IR --model cycle_gan --batch_size 8 --load_size 2560 --preprocess scale_width_and_crop --crop_size 360 --gpu_ids 0,1,2,3,4,5,6,7 --norm instance
2. python test.py --dataroot single/testIR --name test_IR --model cycle_gan --preprocess none --no_dropout