art-DCGAN
art-DCGAN copied to clipboard
robbiebarrat
What are the training parameters for pre-trained networks?
I've been training for cubism style with 500 images (I know I should have around ~3k but there's not enough sample) with nz=300 ndf=20 ngf=100 (if I make them higher D or G would fixate on 0) and here are my results after 750, 1500 and 2250 iterations respectively:
Any suggestions for parameter tuning?
first off - try horizontally mirroring the images to get more data.
your network is spitting out the same images over and over (or variants of them) - which usually means that the generator has outsmarted the discriminator. try either reducing the number of filters in the generator, or increasing the ones in the discriminator.
or - use a pre-trained network (like the landscapes one) - and train that on cubism for a few epochs. you will definitely get more stable results that way.
I mirrored the images like you said, then trained the network using ndf=50 ngf=125:
Somewhere past the 750th iteration one of the networks reached 0 and the other 27 so I stopped the training. Is this because my data sample is still low? Or did it get solved in 750 iteration - is that ever supposed to happen?
I tried again using ndf=45 ngf=120:
your network is spitting out the same images over and over (or variants of them) - which usually means that the generator has outsmarted the discriminator.
at this point, if I modify ndf or ngf training is stuck after second iteration:
ndf : 75
ngf : 140
Epoch: [1][ 0 / 15] Time: 11.247 DataTime: 0.001 Err_G: 1.9977 Err_D: 2.3039
Epoch: [1][ 1 / 15] Time: 0.934 DataTime: 0.005 Err_G: 0.0000 Err_D: 17.6696
Epoch: [1][ 2 / 15] Time: 1.540 DataTime: 0.001 Err_G: 3.0010 Err_D: 0.4860
Epoch: [1][ 3 / 15] Time: 1.538 DataTime: 0.001 Err_G: 0.0545 Err_D: 5.4351
Epoch: [1][ 4 / 15] Time: 1.537 DataTime: 0.001 Err_G: 11.4275 Err_D: 2.3259
Epoch: [1][ 5 / 15] Time: 1.542 DataTime: 0.001 Err_G: 0.0000 Err_D: 14.4293
Epoch: [1][ 6 / 15] Time: 1.551 DataTime: 0.001 Err_G: 11.4585 Err_D: 1.7864
Epoch: [1][ 7 / 15] Time: 1.541 DataTime: 0.001 Err_G: 0.0009 Err_D: 9.7629
Epoch: [1][ 8 / 15] Time: 1.540 DataTime: 0.001 Err_G: 19.6811 Err_D: 1.5082
Epoch: [1][ 9 / 15] Time: 2.630 DataTime: 0.001 Err_G: 0.0101 Err_D: 6.8360
Epoch: [1][ 10 / 15] Time: 0.919 DataTime: 0.001 Err_G: 20.2226 Err_D: 2.1656
Epoch: [1][ 11 / 15] Time: 1.541 DataTime: 0.001 Err_G: 0.0001 Err_D: 12.2572
Epoch: [1][ 12 / 15] Time: 1.549 DataTime: 0.001 Err_G: 20.4725 Err_D: 1.3243
Epoch: [1][ 13 / 15] Time: 1.549 DataTime: 0.001 Err_G: 0.0028 Err_D: 9.1036
Epoch: [1][ 14 / 15] Time: 1.543 DataTime: 0.001 Err_G: 25.9297 Err_D: 2.1749
Epoch: [1][ 15 / 15] Time: 1.538 DataTime: 0.001 Err_G: 5.9669 Err_D: 0.7090
End of epoch 1 / 2500 Time Taken: 34.242
Epoch: [2][ 0 / 15] Time: 1.537 DataTime: 0.001 Err_G: 0.0000 Err_D: 27.7518
Epoch: [2][ 1 / 15] Time: 1.476 DataTime: 0.001 Err_G: 0.0000 Err_D: 27.7053
Epoch: [2][ 2 / 15] Time: 1.477 DataTime: 0.001 Err_G: 0.0000 Err_D: 27.6438
Epoch: [2][ 3 / 15] Time: 1.474 DataTime: 0.001 Err_G: 0.0000 Err_D: 27.6322
Epoch: [2][ 4 / 15] Time: 1.475 DataTime: 0.001 Err_G: 0.0000 Err_D: 27.6385
Epoch: [2][ 5 / 15] Time: 1.477 DataTime: 0.001 Err_G: 0.0000 Err_D: 27.6317
Epoch: [2][ 6 / 15] Time: 1.479 DataTime: 0.001 Err_G: 0.0000 Err_D: 27.6500
Epoch: [2][ 7 / 15] Time: 1.472 DataTime: 0.001 Err_G: 0.0000 Err_D: 27.6529
Epoch: [2][ 8 / 15] Time: 1.473 DataTime: 0.001 Err_G: 0.0000 Err_D: 27.6390
Epoch: [2][ 9 / 15] Time: 2.478 DataTime: 0.001 Err_G: 0.0000 Err_D: 27.6334
Epoch: [2][ 10 / 15] Time: 0.920 DataTime: 0.001 Err_G: 0.0000 Err_D: 27.6319
Epoch: [2][ 11 / 15] Time: 1.477 DataTime: 0.001 Err_G: 0.0000 Err_D: 27.6328
Epoch: [2][ 12 / 15] Time: 1.473 DataTime: 0.001 Err_G: 0.0000 Err_D: 27.6319
Epoch: [2][ 13 / 15] Time: 1.479 DataTime: 0.001 Err_G: 0.0000 Err_D: 27.6314
Epoch: [2][ 14 / 15] Time: 1.477 DataTime: 0.001 Err_G: 0.0000 Err_D: 27.6318
Epoch: [2][ 15 / 15] Time: 1.484 DataTime: 0.001 Err_G: 0.0000 Err_D: 27.6339
ndf : 55
ngf : 120
Epoch: [1][ 0 / 15] Time: 9.819 DataTime: 0.001 Err_G: 0.6422 Err_D: 2.2376
Epoch: [1][ 1 / 15] Time: 0.598 DataTime: 0.001 Err_G: 0.0480 Err_D: 4.9154
Epoch: [1][ 2 / 15] Time: 1.001 DataTime: 0.001 Err_G: 3.0255 Err_D: 2.1711
Epoch: [1][ 3 / 15] Time: 0.979 DataTime: 0.001 Err_G: 0.0001 Err_D: 11.6199
Epoch: [1][ 4 / 15] Time: 0.986 DataTime: 0.001 Err_G: 1.9036 Err_D: 1.5011
Epoch: [1][ 5 / 15] Time: 0.970 DataTime: 0.001 Err_G: 0.0116 Err_D: 6.4144
Epoch: [1][ 6 / 15] Time: 0.978 DataTime: 0.001 Err_G: 8.6712 Err_D: 1.4926
Epoch: [1][ 7 / 15] Time: 0.976 DataTime: 0.001 Err_G: 0.0000 Err_D: 12.0402
Epoch: [1][ 8 / 15] Time: 0.983 DataTime: 0.001 Err_G: 4.9386 Err_D: 1.1580
Epoch: [1][ 9 / 15] Time: 1.763 DataTime: 0.001 Err_G: 0.0001 Err_D: 11.5025
Epoch: [1][ 10 / 15] Time: 0.583 DataTime: 0.001 Err_G: 9.6556 Err_D: 1.3580
Epoch: [1][ 11 / 15] Time: 0.976 DataTime: 0.001 Err_G: 0.0019 Err_D: 7.9405
Epoch: [1][ 12 / 15] Time: 0.978 DataTime: 0.001 Err_G: 14.5594 Err_D: 1.2251
Epoch: [1][ 13 / 15] Time: 0.983 DataTime: 0.001 Err_G: 0.0127 Err_D: 7.0154
Epoch: [1][ 14 / 15] Time: 0.986 DataTime: 0.001 Err_G: 15.0553 Err_D: 0.7528
Epoch: [1][ 15 / 15] Time: 0.980 DataTime: 0.001 Err_G: 0.0270 Err_D: 6.3222
End of epoch 1 / 2500 Time Taken: 24.540
Epoch: [2][ 0 / 15] Time: 0.978 DataTime: 0.001 Err_G: 17.3309 Err_D: 1.5832
Epoch: [2][ 1 / 15] Time: 0.979 DataTime: 0.001 Err_G: 0.0668 Err_D: 4.5823
Epoch: [2][ 2 / 15] Time: 0.977 DataTime: 0.001 Err_G: 16.7114 Err_D: 1.4077
Epoch: [2][ 3 / 15] Time: 0.992 DataTime: 0.001 Err_G: 0.0003 Err_D: 9.9892
Epoch: [2][ 4 / 15] Time: 0.979 DataTime: 0.001 Err_G: 18.4478 Err_D: 1.3203
Epoch: [2][ 5 / 15] Time: 0.980 DataTime: 0.001 Err_G: 0.3714 Err_D: 2.5584
Epoch: [2][ 6 / 15] Time: 0.997 DataTime: 0.001 Err_G: 8.9893 Err_D: 1.0589
Epoch: [2][ 7 / 15] Time: 0.987 DataTime: 0.001 Err_G: 0.0000 Err_D: 27.0925
Epoch: [2][ 8 / 15] Time: 0.978 DataTime: 0.001 Err_G: 0.0000 Err_D: 26.1768
Epoch: [2][ 9 / 15] Time: 1.763 DataTime: 0.001 Err_G: 0.0004 Err_D: 9.1433
Epoch: [2][ 10 / 15] Time: 0.577 DataTime: 0.001 Err_G: 27.6180 Err_D: 0.5351
Epoch: [2][ 11 / 15] Time: 0.990 DataTime: 0.001 Err_G: 27.6310 Err_D: 0.7237
Epoch: [2][ 12 / 15] Time: 0.982 DataTime: 0.001 Err_G: 27.6310 Err_D: 0.3762
Epoch: [2][ 13 / 15] Time: 0.979 DataTime: 0.001 Err_G: 27.6310 Err_D: 0.0788
Epoch: [2][ 14 / 15] Time: 0.978 DataTime: 0.001 Err_G: 27.6310 Err_D: 0.1221
Epoch: [2][ 15 / 15] Time: 0.980 DataTime: 0.001 Err_G: 27.6310 Err_D: 0.0090