Same output for all inputs with test.lua

[nk-dev0]

I'm training a model using my own data, and I've made sure to recreate the folder structure required and ran combine_A_and_B.py to create training data. In my both my test and val folders I have 1000 imageA/imageB pairs.

When I run DATA_ROOT=</my/data/root/> name=<my_expt> which_direction=AtoB th test.lua (with my actual experiment name and data root), it produces the correct folder output with latest_net_G_val being the top level folder, and images and index.html

However, every single output image is the same and doesn't correspond at all to the input or target images, but it does look like it's doing a decent job at segmenting some unknown input image from the dataset. There is some very slight variation between each output image, so it seems it's just getting the same single input into the network each time (?) Any help would be greatly appreciated.

[junyanz]

Did you see the same phenomena during your training? Are you able to run the train/test code on the facade dataset without problems?

[nk-dev0]

Yes, I did see the same outputs during training. All of the sample tests had the same output even though the inputs were different. I have not yet tried the facade dataset.

[junyanz]

I recommend you try to run the code on facade dataset first, and also double-check if your generated training/val set is correct (/my/data/root/)

[nk-dev0]

I've found that when the train set size is larger than the save_latest_freq or save_display_freq in train.lua, the outputs will all be identical no matter the input image. My original train set size was ~7700 with the default save settings of 5000 iterations, and I was seeing this error, but when I reduced to ~4000 with the same save settings of 5000, it worked fine.