Resuming training does not work

[zzh910]

Hi,

I set the parameters as follows:

#!/bin/bash

./Caffe_BEGAN ^ --train=1 ^ --weights_gen= ^ --weights_discr= ^ --z_dim=128 ^ --h_dim=128 ^ --hidden_dim=64 ^ --image_size=128 ^ --solver_gen=solver_generator.prototxt ^ --solver_discr=solver_discriminator.prototxt ^ --batch_size=16 ^ --snapshot_generator=BEGAN_Generator_iter_500000.solverstate ^ --snapshot_discriminator=BEGAN_Discriminator_iter_500000.solverstate ^ --snapshot_k_t=k_t_iter_500000.txt ^ --preview_generator=preview_values.csv ^ --number_batch_loaded=125 ^ --start_epoch=0 ^ --end_epoch=25 ^ --training_dataset=../Data/CelebA

I gave *.sloverstate files and corresponding t_t.

The program always generates an image before the training, so I checked the Generation_epoch_0.bmp, it's pure noise. I thought the first image generated should be faces since I have resumed training from iteration 500000.

Did I set the parameters wrong?

Thank you, Sebastian

[adepierre]

Hi,

That's a bit strange. I don't see anything wrong in your parameters. One quick thing to check is if the log.csv file is erased. If the solvers are correctly restored, it should append the new logs and not overwrite the file (https://github.com/adepierre/Caffe_BEGAN/blob/master/src/NN_Agent.cpp#L84).

You could also set the weights_gen and weights_discr flags with the corresponding .caffemodel to see if it helps, but as far as I remember the solverstate Restore function should load them.

I'm sorry I'm not able to help you more but this is quite an old code and I don't perfectly remember all the details ^_^ If you are really in trouble with this one I'll try to recompile it and check if there is a bug