Inconsistency in training loss (300W-LP) and testing loss (AFLW2000). What should be the convergence criterion and when to save best model?

[GKG1312]

Hi @natanielruiz @tfygg , I am training the model again on 300W-LP dataset with filtered filenames. There is high fluctuation in the training loss as mentioned in previous issues as well #6 and #10. Even though for some iteration the losses are very low:

Epoch [25/25], Iter [600/3825] Losses: Yaw 2.5382, Pitch 25.2214, Roll 18.6293 Epoch [25/25], Iter [700/3825] Losses: Yaw 3.4427, Pitch 56.4101, Roll 60.4185 Epoch [25/25], Iter [800/3825] Losses: Yaw 3.8120, Pitch 10.9580, Roll 12.5700 Epoch [25/25], Iter [900/3825] Losses: Yaw 6.2587, Pitch 36.2516, Roll 29.5404 Epoch [25/25], Iter [1000/3825] Losses: Yaw 3.9143, Pitch 13.5918, Roll 11.6238 Epoch [25/25], Iter [1100/3825] Losses: Yaw 2.8406, Pitch 16.2069, Roll 11.7216 Epoch [25/25], Iter [1200/3825] Losses: Yaw 3.1640, Pitch 6.9615, Roll 3.9374 Epoch [25/25], Iter [1300/3825] Losses: Yaw 4.6969, Pitch 8.0815, Roll 9.0429 Epoch [25/25], Iter [1400/3825] Losses: Yaw 3.1008, Pitch 6.8233, Roll 4.4145 Epoch [25/25], Iter [1500/3825] Losses: Yaw 3.5320, Pitch 53.3095, Roll 41.4802 Epoch [25/25], Iter [1600/3825] Losses: Yaw 3.7685, Pitch 7.2890, Roll 8.7627 Epoch [25/25], Iter [1700/3825] Losses: Yaw 3.2166, Pitch 19.6407, Roll 12.9610 Epoch [25/25], Iter [1800/3825] Losses: Yaw 3.6263, Pitch 6.8446, Roll 5.8751 Epoch [25/25], Iter [1900/3825] Losses: Yaw 3.7254, Pitch 12.2385, Roll 9.0497 Epoch [25/25], Iter [2000/3825] Losses: Yaw 4.3334, Pitch 10.8476, Roll 4.3712 Epoch [25/25], Iter [2100/3825] Losses: Yaw 4.8823, Pitch 13.0971, Roll 17.6704 Epoch [25/25], Iter [2200/3825] Losses: Yaw 2.9647, Pitch 5.1831, Roll 5.9912 Epoch [25/25], Iter [2300/3825] Losses: Yaw 2.6243, Pitch 20.3848, Roll 10.7074 Epoch [25/25], Iter [2400/3825] Losses: Yaw 4.3780, Pitch 16.6918, Roll 10.1041 Epoch [25/25], Iter [2500/3825] Losses: Yaw 2.6419, Pitch 29.8599, Roll 23.3731 Epoch [25/25], Iter [2600/3825] Losses: Yaw 3.0582, Pitch 23.6246, Roll 15.0430 Epoch [25/25], Iter [2700/3825] Losses: Yaw 4.5449, Pitch 11.4036, Roll 9.0669 Epoch [25/25], Iter [2800/3825] Losses: Yaw 3.3777, Pitch 6.4258, Roll 4.7266 Epoch [25/25], Iter [2900/3825] Losses: Yaw 4.5212, Pitch 8.0623, Roll 5.5993 Epoch [25/25], Iter [3000/3825] Losses: Yaw 3.5405, Pitch 11.6594, Roll 9.8117 Epoch [25/25], Iter [3100/3825] Losses: Yaw 2.8780, Pitch 10.0156, Roll 9.4295 Epoch [25/25], Iter [3200/3825] Losses: Yaw 3.9240, Pitch 8.4466, Roll 4.5813 Epoch [25/25], Iter [3300/3825] Losses: Yaw 4.6378, Pitch 8.8315, Roll 8.9284

While testing the model on AFLW2000 I am getting slight high error in Yaw. Test error in degrees of the model on the 1969 test images. Yaw: 13.6368, Pitch: 7.7751, Roll: 6.1729 I am saving my model based on best iteration where all errors are minimum.

Can you help me to find out the reason. Following is the command I am using to train the model (I am continuing training over a saved model): train_hopenet.py --data_dir ".\300W_LP" --filename_list "300W_LP_filename_filtered.txt" --snapshot "Pruned_Hopenet_0.5.pth" --batch_size 32 --dataset "Pose_300W_LP" --num_epochs 25 --alpha 1 --output_string "prunedReTrain_0.5_1st" --lr 0.00001

Note : Just my observation, the error for Yaw is less as compare to others while training. However at time of testing on AFLW2000, error in Yaw is largest. Is this because of the dataset? Did you observed anything like that when you trained and tested your model?