Can't improve rotation error when training with my custom data

[CongDuy038]

I'm having a problem with training on my custom dataset. I use the weight pretrained file efficientdet-d2.h5 It seems that the translation error has been improved very well. However, the rotation error is quite high and does not decrease. Here is the output of the epochs I train:

mAP: 1.0000 ADD: 0.7526 ADD-S: 0.7555 5cm_5degree: 0.0000 TranslationErrorMean_in_mm: 9.8703 TranslationErrorStd_in_mm: 5.0323 RotationErrorMean_in_degree: 111.7360 RotationErrorStd_in_degree: 36.6118 2D-Projection: 0.1885 Summed_Translation_Rotation_Error: 163.2504 ADD(-S): 0.7526 AveragePointDistanceMean_in_mm: 9.8673 AveragePointDistanceStd_in_mm: 5.0325 AverageSymmetricPointDistanceMean_in_mm: 9.8284 AverageSymmetricPointDistanceStd_in_mm: 5.0331 MixedAveragePointDistanceMean_in_mm: 9.8673 MixedAveragePointDistanceStd_in_mm: 5.0325 Epoch 00001: ADD improved from -inf to 0.75258, saving model to checkpoints/02_10_2022_12_17_12/object_2/phi_0_linemod_best_ADD.h5 1000/1000 [==============================] - 1124s 1s/step - loss: 1.5798 - classification_loss: 0.6198 - regression_loss: 0.0557 - transformation_loss: 45.2144 Epoch 2/4 Running network: 100% (679 of 679) |######| Elapsed Time: 0:00:36 Time: 0:00:36 Parsing annotations: 100% (679 of 679) |##| Elapsed Time: 0:00:00 Time: 0:00:00 100% 679/679 [00:08<00:00, 77.82it/s] 679 instances of class object with average precision: 1.0000 679 instances of class object with ADD accuracy: 0.1532 679 instances of class object with ADD-S-Accuracy: 0.1561 679 instances of class object with 5cm-5degree-Accuracy: 0.0000 class object with Translation Differences in mm: Mean: 19.1808 and Std: 6.8850 class object with Rotation Differences in degree: Mean: 127.1700 and Std: 33.4669 679 instances of class object with 2d-projection-Accuracy: 0.1915 679 instances of class object with ADD(-S)-Accuracy: 0.1532 class object with Transformed Point Distances in mm: Mean: 19.1779 and Std: 6.8854 class object with Transformed Symmetric Point Distances in mm: Mean: 19.1396 and Std: 6.8854 class object with Mixed Transformed Point Distances in mm: Mean: 19.1779 and Std: 6.8854 mAP: 1.0000 ADD: 0.1532 ADD-S: 0.1561 5cm_5degree: 0.0000 TranslationErrorMean_in_mm: 19.1808 TranslationErrorStd_in_mm: 6.8850 RotationErrorMean_in_degree: 127.1700 RotationErrorStd_in_degree: 33.4669 2D-Projection: 0.1915 Summed_Translation_Rotation_Error: 186.7028 ADD(-S): 0.1532 AveragePointDistanceMean_in_mm: 19.1779 AveragePointDistanceStd_in_mm: 6.8854 AverageSymmetricPointDistanceMean_in_mm: 19.1396 AverageSymmetricPointDistanceStd_in_mm: 6.8854 MixedAveragePointDistanceMean_in_mm: 19.1779 MixedAveragePointDistanceStd_in_mm: 6.8854

Epoch 00002: ADD did not improve from 0.75258 1000/1000 [==============================] - 318s 318ms/step - loss: 0.5754 - classification_loss: 0.0534 - regression_loss: 0.0288 - transformation_loss: 24.6600 Epoch 3/4 Running network: 100% (679 of 679) |######| Elapsed Time: 0:00:36 Time: 0:00:36 Parsing annotations: 100% (679 of 679) |##| Elapsed Time: 0:00:00 Time: 0:00:00 100% 679/679 [00:08<00:00, 80.08it/s] 679 instances of class object with average precision: 1.0000 679 instances of class object with ADD accuracy: 0.3888 679 instances of class object with ADD-S-Accuracy: 0.3932 679 instances of class object with 5cm-5degree-Accuracy: 0.0000 class object with Translation Differences in mm: Mean: 13.7990 and Std: 4.9412 class object with Rotation Differences in degree: Mean: 115.2414 and Std: 40.1505 679 instances of class object with 2d-projection-Accuracy: 0.2018 679 instances of class object with ADD(-S)-Accuracy: 0.3888 class object with Transformed Point Distances in mm: Mean: 13.7954 and Std: 4.9409 class object with Transformed Symmetric Point Distances in mm: Mean: 13.7568 and Std: 4.9408 class object with Mixed Transformed Point Distances in mm: Mean: 13.7954 and Std: 4.9409 mAP: 1.0000 ADD: 0.3888 ADD-S: 0.3932 5cm_5degree: 0.0000 TranslationErrorMean_in_mm: 13.7990 TranslationErrorStd_in_mm: 4.9412 RotationErrorMean_in_degree: 115.2414 RotationErrorStd_in_degree: 40.1505 2D-Projection: 0.2018 Summed_Translation_Rotation_Error: 174.1321 ADD(-S): 0.3888 AveragePointDistanceMean_in_mm: 13.7954 AveragePointDistanceStd_in_mm: 4.9409 AverageSymmetricPointDistanceMean_in_mm: 13.7568 AverageSymmetricPointDistanceStd_in_mm: 4.9408 MixedAveragePointDistanceMean_in_mm: 13.7954 MixedAveragePointDistanceStd_in_mm: 4.9409

Epoch 00003: ADD did not improve from 0.75258 1000/1000 [==============================] - 313s 313ms/step - loss: 0.5011 - classification_loss: 0.0364 - regression_loss: 0.0232 - transformation_loss: 22.0763 Epoch 4/4 Running network: 100% (679 of 679) |######| Elapsed Time: 0:00:37 Time: 0:00:37 Parsing annotations: 100% (679 of 679) |##| Elapsed Time: 0:00:00 Time: 0:00:00 100% 679/679 [00:08<00:00, 77.52it/s] 679 instances of class object with average precision: 1.0000 679 instances of class object with ADD accuracy: 0.5626 679 instances of class object with ADD-S-Accuracy: 0.5700 679 instances of class object with 5cm-5degree-Accuracy: 0.0000 class object with Translation Differences in mm: Mean: 12.2080 and Std: 6.1116 class object with Rotation Differences in degree: Mean: 130.5616 and Std: 35.5283 679 instances of class object with 2d-projection-Accuracy: 0.4227 679 instances of class object with ADD(-S)-Accuracy: 0.5626 class object with Transformed Point Distances in mm: Mean: 12.2090 and Std: 6.1138 class object with Transformed Symmetric Point Distances in mm: Mean: 12.1643 and Std: 6.1119 class object with Mixed Transformed Point Distances in mm: Mean: 12.2090 and Std: 6.1138 mAP: 1.0000 ADD: 0.5626 ADD-S: 0.5700 5cm_5degree: 0.0000 TranslationErrorMean_in_mm: 12.2080 TranslationErrorStd_in_mm: 6.1116 RotationErrorMean_in_degree: 130.5616 RotationErrorStd_in_degree: 35.5283 2D-Projection: 0.4227 Summed_Translation_Rotation_Error: 184.4095 ADD(-S): 0.5626 AveragePointDistanceMean_in_mm: 12.2090 AveragePointDistanceStd_in_mm: 6.1138 AverageSymmetricPointDistanceMean_in_mm: 12.1643 AverageSymmetricPointDistanceStd_in_mm: 6.1119 MixedAveragePointDistanceMean_in_mm: 12.2090 MixedAveragePointDistanceStd_in_mm: 6.1138

Epoch 00004: ADD did not improve from 0.75258 1000/1000 [==============================] - 317s 317ms/step - loss: 0.4195 - classification_loss: 0.0321 - regression_loss: 0.0201 - transformation_loss: 18.3656

==================================================================================

This is the image when I run the evaluate. It detects objects well, but the rotation doesn't seem to match the ground truth box.

Tell me how I can reduce the rotation error!

[8umpk1n]

hello! I want to ask some questions about the production of datasets, can you give me some help?

[Hijsm]

Are you sure your annotation data is alright? The ground truth in your images does not seem well fitted around your object, nor is it consistently off. I think it would help a lot if you had the right dataset.

[madhanuman]

Were you able to solve this @CongDuy038? I am standing infront of the exact same problem and i can not see where my mistake is...

[SMhang]

May I ask if you have resolved it? I have encountered the same problem as you