The regression loss of car.waymo.StarNetVehicle is always zero

[DapengFeng]

Hi, I use all the data with camera labels in the waymo open dataset v1.0.0 to train the StarNet model, but after 16924 steps, the logger always print out the regression loss is zero from the begin of training stage.

I0329 10:46:45.515985 140700620408576 summary_utils.py:342] Steps/second: 0.068840, Examples/second: 0.275359
I0329 10:46:45.518149 140700620408576 trainer.py:522] step: 16918, steps/sec: 0.07, examples/sec: 0.28 error/center_distance:0 error/height:-0 error/length:-0 error/rotation_deg:0 error/width:-0 grad_norm/all/loss:0.00086083304 grad_scale_all/loss:1 has_nan_or_inf/loss:0 loss:1.1038595e-05 loss/classification:1.1038595e-05 loss/regression:0 loss/regression/corner:0 loss/regression/dim:0 loss/regression/loc:0 loss/regression/rot:0 num_samples_in_batch:4 var_norm/all/loss:7.3228803
I0329 10:47:04.314408 140700620408576 summary_utils.py:342] Steps/second: 0.068837, Examples/second: 0.275348
I0329 10:47:04.317717 140700620408576 trainer.py:522] step: 16919, steps/sec: 0.07, examples/sec: 0.28 error/center_distance:0 error/height:-0 error/length:-0 error/rotation_deg:0 error/width:-0 grad_norm/all/loss:0.0015155645 grad_scale_all/loss:1 has_nan_or_inf/loss:0 loss:1.0980473e-05 loss/classification:1.0980473e-05 loss/regression:0 loss/regression/corner:0 loss/regression/dim:0 loss/regression/loc:0 loss/regression/rot:0 num_samples_in_batch:4 var_norm/all/loss:7.3202777
I0329 10:47:12.916368 140700620408576 summary_utils.py:342] Steps/second: 0.068839, Examples/second: 0.275357
I0329 10:47:12.919012 140700620408576 trainer.py:522] step: 16920, steps/sec: 0.07, examples/sec: 0.28 error/center_distance:0 error/height:-0 error/length:-0 error/rotation_deg:0 error/width:-0 grad_norm/all/loss:0.00091527984 grad_scale_all/loss:1 has_nan_or_inf/loss:0 loss:1.041985e-05 loss/classification:1.041985e-05 loss/regression:0 loss/regression/corner:0 loss/regression/dim:0 loss/regression/loc:0 loss/regression/rot:0 num_samples_in_batch:4 var_norm/all/loss:7.3176508
I0329 10:47:28.270684 140700620408576 summary_utils.py:342] Steps/second: 0.068838, Examples/second: 0.275354
I0329 10:47:28.275026 140700620408576 trainer.py:522] step: 16921, steps/sec: 0.07, examples/sec: 0.28 error/center_distance:0 error/height:-0 error/length:-0 error/rotation_deg:0 error/width:-0 grad_norm/all/loss:0.0010528042 grad_scale_all/loss:1 has_nan_or_inf/loss:0 loss:9.7481316e-06 loss/classification:9.7481316e-06 loss/regression:0 loss/regression/corner:0 loss/regression/dim:0 loss/regression/loc:0 loss/regression/rot:0 num_samples_in_batch:4 var_norm/all/loss:7.315033
I0329 10:47:41.014512 140700620408576 summary_utils.py:342] Steps/second: 0.068838, Examples/second: 0.275351
I0329 10:47:41.026140 140700620408576 trainer.py:522] step: 16922, steps/sec: 0.07, examples/sec: 0.28 error/center_distance:0 error/height:-0 error/length:-0 error/rotation_deg:0 error/width:-0 grad_norm/all/loss:0.0021238418 grad_scale_all/loss:1 has_nan_or_inf/loss:0 loss:9.9192612e-06 loss/classification:9.9192612e-06 loss/regression:0 loss/regression/corner:0 loss/regression/dim:0 loss/regression/loc:0 loss/regression/rot:0 num_samples_in_batch:4 var_norm/all/loss:7.3124075
I0329 10:47:59.705081 140700620408576 summary_utils.py:342] Steps/second: 0.068836, Examples/second: 0.275344
I0329 10:47:59.708905 140700620408576 trainer.py:522] step: 16923, steps/sec: 0.07, examples/sec: 0.28 error/center_distance:0 error/height:-0 error/length:-0 error/rotation_deg:0 error/width:-0 grad_norm/all/loss:0.0011530347 grad_scale_all/loss:1 has_nan_or_inf/loss:0 loss:1.0637985e-05 loss/classification:1.0637985e-05 loss/regression:0 loss/regression/corner:0 loss/regression/dim:0 loss/regression/loc:0 loss/regression/rot:0 num_samples_in_batch:4 var_norm/all/loss:7.3097687
I0329 10:48:11.894471 140700620408576 summary_utils.py:342] Steps/second: 0.068835, Examples/second: 0.275341
I0329 10:48:11.898586 140700620408576 trainer.py:522] step: 16924, steps/sec: 0.07, examples/sec: 0.28 error/center_distance:0 error/height:-0 error/length:-0 error/rotation_deg:0 error/width:-0 grad_norm/all/loss:0.0011890592 grad_scale_all/loss:1 has_nan_or_inf/loss:0 loss:1.0627678e-05 loss/classification:1.0627678e-05 loss/regression:0 loss/regression/corner:0 loss/regression/dim:0 loss/regression/loc:0 loss/regression/rot:0 num_samples_in_batch:4 var_norm/all/loss:7.3071561
I0329 10:48:23.664032 140700620408576 summary_utils.py:342] Steps/second: 0.068837, Examples/second: 0.275348
I0329 10:48:23.666607 140700620408576 trainer.py:522] step: 16925, steps/sec: 0.07, examples/sec: 0.28 error/center_distance:0 error/height:-0 error/length:-0 error/rotation_deg:0 error/width:-0 grad_norm/all/loss:0.0008956138 grad_scale_all/loss:1 has_nan_or_inf/loss:0 loss:1.2235249e-05 loss/classification:1.2235249e-05 loss/regression:0 loss/regression/corner:0 loss/regression/dim:0 loss/regression/loc:0 loss/regression/rot:0 num_samples_in_batch:4 var_norm/all/loss:7.3045363
I0329 10:48:37.568562 140700620408576 summary_utils.py:342] Steps/second: 0.068835, Examples/second: 0.275341
I0329 10:48:37.571758 140700620408576 trainer.py:522] step: 16926, steps/sec: 0.07, examples/sec: 0.28 error/center_distance:0 error/height:-0 error/length:-0 error/rotation_deg:0 error/width:-0 grad_norm/all/loss:0.0011131507 grad_scale_all/loss:1 has_nan_or_inf/loss:0 loss:1.072419e-05 loss/classification:1.072419e-05 loss/regression:0 loss/regression/corner:0 loss/regression/dim:0 loss/regression/loc:0 loss/regression/rot:0 num_samples_in_batch:4 var_norm/all/loss:7.3019686
I0329 10:48:50.108403 140700620408576 summary_utils.py:342] Steps/second: 0.068838, Examples/second: 0.275351
I0329 10:48:50.117931 140700620408576 trainer.py:522] step: 16927, steps/sec: 0.07, examples/sec: 0.28 error/center_distance:0 error/height:-0 error/length:-0 error/rotation_deg:0 error/width:-0 grad_norm/all/loss:0.00042419846 grad_scale_all/loss:1 has_nan_or_inf/loss:0 loss:1.3342853e-05 loss/classification:1.3342853e-05 loss/regression:0 loss/regression/corner:0 loss/regression/dim:0 loss/regression/loc:0 loss/regression/rot:0 num_samples_in_batch:4 var_norm/all/loss:7.2993846
I0329 10:49:03.650874 140700620408576 summary_utils.py:342] Steps/second: 0.068835, Examples/second: 0.275340
I0329 10:49:03.653239 140700620408576 trainer.py:522] step: 16928, steps/sec: 0.07, examples/sec: 0.28 error/center_distance:0 error/height:-0 error/length:-0 error/rotation_deg:0 error/width:-0 grad_norm/all/loss:0.00025687742 grad_scale_all/loss:1 has_nan_or_inf/loss:0 loss:9.532584e-06 loss/classification:9.532584e-06 loss/regression:0 loss/regression/corner:0 loss/regression/dim:0 loss/regression/loc:0 loss/regression/rot:0 num_samples_in_batch:4 var_norm/all/loss:7.2968655