Huge training loss while fine-tuning a pretrained SSD checkpoint on VOC 2007 dataset

[derekhh]

Description

tl;dr: The pre-trained SSD checkpoint has a huge loss while training on the VOC2007 train + val dataset and it doesn't seem to be anywhere near convergence.

I was trying to use the pre-trained SSD checkpoint here to fine-tune the VOC 2007 train+val dataset. I understand this doesn't make real sense as the SSD model is already pre-trained on the VOC dataset but I want to try it out for a sanity check after cloning the repo.

Initially I followed this training script. The loss seems pretty huge ever after the initial step - which seems very weird. Then I noticed the learning rate probably is too high for fine-tuning and I changed it to something like 1e-5 and 1e-6. Still I'm observing huge training loss and it doesn't seem to be converging.

DATASET_DIR=./tfrecords
TRAIN_DIR=./logs/
CHECKPOINT_PATH=./checkpoints/ssd_300_vgg.ckpt
python train_ssd_network.py \
    --train_dir=${TRAIN_DIR} \
    --dataset_dir=${DATASET_DIR} \
    --dataset_name=pascalvoc_2012 \
    --dataset_split_name=train \
    --model_name=ssd_300_vgg \
    --checkpoint_path=${CHECKPOINT_PATH} \
    --save_summaries_secs=60 \
    --save_interval_secs=600 \
    --weight_decay=0.0005 \
    --optimizer=adam \
    --learning_rate=0.001 \
    --batch_size=32

I've then evaluated the SSD checkpoint quality on both the VOC07 train + val dataset and VOC07 test dataset to make sure the quality of the checkpoint is OK. I'm getting decent mAP values on both datasets so this confuses me a lot as I feel this should mean the loss is low while training on VOC07 train+val.

I tensorflow/core/kernels/logging_ops.cc:79] AP_VOC07/mAP[0.81998744666155454]
I tensorflow/core/kernels/logging_ops.cc:79] AP_VOC12/mAP[0.85548588208867282]
INFO:tensorflow:Finished evaluation at 2017-04-19-18:02:18
Time spent : 301.644 seconds.
Time spent per BATCH: 0.481 seconds.

Can @balancap help take a look? And thanks again for sharing this repo with us!

[villanuevab]

I am getting similar results, but with a much higher training loss that never drops below 40 in some cases (even after running ~100k steps).

I am using all default scripts, running with:

python train_ssd_network.py --train_dir=/summary/ssd_300_pascal --dat_dir=/home/blanca/my_data/voc2007_tfrecords/ --checkpoint_path=./checkpoints/ssd_300_vgg.ckpt --checkpoint_exclude_scopes=ssd_300_vgg/block4_box,ssd_300_vgg/block7_box,ssd_300_vgg/block8_box,ssd_300_vgg/block9_box,ssd_300_vgg/block10_box,ssd_300_vgg/block11_box --dataset_name=pascalvoc_2007 --dataset_split_name=train --model_name=ssd_300_vgg --save_summaries_secs=60 --save_interval_secs=60 --weight_decay=0.0005 --learning_rate=0.001 --learning_rate_decay_factor=0.96 --batch_size=32 --gpu_memory_fraction=0.8 --num_classes=21

And yet I still achieve a high mAP on the training set (this is for sanity check purposes. I have not tested on Pascal VOC test data yet):

INFO:tensorflow:Evaluation [5011/5011]
I tensorflow/core/kernels/logging_ops.cc:79] AP_VOC07/mAP[0.82002322122807825]
I tensorflow/core/kernels/logging_ops.cc:79] AP_VOC12/mAP[0.855447542894266]
INFO:tensorflow:Finished evaluation at 2017-05-03-16:47:24
Time spent : 430.635 seconds.
Time spent per BATCH: 0.086 seconds.

Has anyone solved this issue?

[nlngh]

In general, regularization losses are not included in model evaluation hence losses in training phase highly overestimate

[qianweilzh]

Description

tl;dr: The pre-trained SSD checkpoint has a huge loss while training on the VOC2007 train + val dataset and it doesn't seem to be anywhere near convergence.

I was trying to use the pre-trained SSD checkpoint here to fine-tune the VOC 2007 train+val dataset. I understand this doesn't make real sense as the SSD model is already pre-trained on the VOC dataset but I want to try it out for a sanity check after cloning the repo.

Initially I followed this training script. The loss seems pretty huge ever after the initial step - which seems very weird. Then I noticed the learning rate probably is too high for fine-tuning and I changed it to something like 1e-5 and 1e-6. Still I'm observing huge training loss and it doesn't seem to be converging.

DATASET_DIR=./tfrecords
TRAIN_DIR=./logs/
CHECKPOINT_PATH=./checkpoints/ssd_300_vgg.ckpt
python train_ssd_network.py \
    --train_dir=${TRAIN_DIR} \
    --dataset_dir=${DATASET_DIR} \
    --dataset_name=pascalvoc_2012 \
    --dataset_split_name=train \
    --model_name=ssd_300_vgg \
    --checkpoint_path=${CHECKPOINT_PATH} \
    --save_summaries_secs=60 \
    --save_interval_secs=600 \
    --weight_decay=0.0005 \
    --optimizer=adam \
    --learning_rate=0.001 \
    --batch_size=32

I've then evaluated the SSD checkpoint quality on both the VOC07 train + val dataset and VOC07 test dataset to make sure the quality of the checkpoint is OK. I'm getting decent mAP values on both datasets so this confuses me a lot as I feel this should mean the loss is low while training on VOC07 train+val.

I tensorflow/core/kernels/logging_ops.cc:79] AP_VOC07/mAP[0.81998744666155454]
I tensorflow/core/kernels/logging_ops.cc:79] AP_VOC12/mAP[0.85548588208867282]
INFO:tensorflow:Finished evaluation at 2017-04-19-18:02:18
Time spent : 301.644 seconds.
Time spent per BATCH: 0.481 seconds.

Can @balancap help take a look? And thanks again for sharing this repo with us!

@derekhh Can you share your eval.sh script? I'm wondering if you evaluated the ckpt before training?