DecaYale
Using pretrained models to continue training on KITTI Odometry datasets meets some problems
First of all, thanks for your kindness of releasing code and the paper is really inspiring. But I meets some problems using pretrained models for continue training using the command below. python -u train_hdf5.py multi_proc_train --config_path config/kitti_train_ours.prototxt --model_dir ./outputs --use_dist True --gpus_per_node 1 --use_apex True --world_size 1 --dist_port 20000 --pretrained_path weights/ours.tckpt --refine False The pretrained model preformance keeps going worse and worse. As we all know the initial test result is test_avg_translation_error=0.028891, but lately the error comes to 0.30968 even higher(0.70748). I think there are some train setting mistakes but I can't find out why. Because I want to use your pretrained model train on other datasets.
Yes, your current configuration might be incorrect. The pretrained model should have been loaded with the parameter --pretrained_path. But with the default configuration file we provided, the training will start from scratch, which means the learning rate schedule will become large first before attenuating. If you want to finetune the model on your own dataset, you may need to downscale the learning rate.
