fine-tuning training prolem

[peternara]

@MhLiao hi, i have a problem during training.

First, I learned with SynthText data. I learned by changing two learning rates with reference to your paper. so, see logs.. (examples) .... Snapshotting to binary proto file models/VGGNet/text/longer_conv_300x300/VGG_text_longer_conv_300x300_iter_119500.caffemodel I0730 17:58:26.258183 53896 sgd_solver.cpp:273] Snapshotting solver state to binary proto file models/VGGNet/text/longer_conv_300x300/VGG_text_longer_conv_300x300_iter_119500.solverstate I0730 17:58:26.434485 53896 solver.cpp:421] Iteration 119500, Testing net (#0) I0730 17:58:26.434622 53896 net.cpp:693] Ignoring source layer mbox_loss I0730 17:58:37.896144 53896 solver.cpp:531] Test net output #0: detection_eval = 0.790573 I0730 17:58:39.164993 53896 solver.cpp:231] Iteration 119500, loss = 1.79239 I0730 17:58:39.165047 53896 solver.cpp:247] Train net output #0: mbox_loss = 1.72402 (* 1 = 1.72402 loss) I0730 17:58:39.165093 53896 sgd_solver.cpp:106] Iteration 119500, lr = 1e-05 I0730 17:58:52.544618 53896 solver.cpp:231] Iteration 119510, loss = 1.88545 I0730 17:58:52.544683 53896 solver.cpp:247] Train net output #0: mbox_loss = 1.12724 (* 1 = 1.12724 loss) I0730 17:58:53.007946 53896 sgd_solver.cpp:106] Iteration 119510, lr = 1e-05 I0730 17:59:06.140127 53896 solver.cpp:231] Iteration 119520, loss = 1.62385 I0730 17:59:06.140287 53896 solver.cpp:247] Train net output #0: mbox_loss = 1.36057 (* 1 = 1.36057 loss) I0730 17:59:06.140353 53896 sgd_solver.cpp:106] Iteration 119520, lr = 1e-05 I0730 17:59:20.802580 53896 solver.cpp:231] Iteration 119530, loss = 1.7999 I0730 17:59:20.802636 53896 solver.cpp:247] Train net output #0: mbox_loss = 1.54816 (* 1 = 1.54816 loss) ,,,

1. At this log, through the model derived from the learning outcomes, we learned again. = (fine-tune)
   • this model is "VGG_text_longer_conv_300x300_iter_119500.caffemodel"
   • i try to 50k model, too. ( VGG_text_longer_conv_300x300_iter_50000.caffemodel)
   • this model' detection_eval is about 0.80. (see above log)
2. at fine-tune trianing, this file modify..
   • pretrain_model = "VGG_text_longer_conv_300x300_iter_50000.caffemodel" in train_icdar13.py

I modified this file and learned again. However, the detection_eval in the process is still staying at a value of 0.2.

Can you guess what your problem is?

[MhLiao]

The detection_eval of finetune is 0.2 or 0.8?

[peternara]

@MhLiao first training is 0.8 and second training(finetuning) is 0.2

[github14343]

I might met the same problem. Did you solve the problem? Thanks!

[github14343]

@peternara