Why do I always get 0% and NaN for each indicator when I test on MOT17 orMOT15？

[cgg99]

Why do I always get 0% and NaN for each indicator when I test on MOT17 orMOT15？ Has anyone encountered this problem? ………………………… …………………………

[hopesala]

I am training my own dataset the train process loss value is nan the command is : python train.py mot --exp_id crowdhuman_dla34 --gpus 0 --batch_size 1 --load_model '../models/ctdet_coco_dla_2x.pth' --num_epochs 60 --lr_step '50' --data_cfg '../src/lib/cfg/satcar.json'

ot/crowdhuman_dla34 |######### | train: [1][2784/9757]|Tot: 0:17:09 |ETA: 0:44:08 |loss nan |hm_loss nan |wh_loss nan |off_loss nan |id_loss nan |Data 0.024s(0.mot/crowdhuman_dla34 |######### | train: [1][2785/9757]|Tot: 0:17:09 |ETA: 0:44:14 |loss nan |hm_loss nan |wh_loss nan |off_loss nan |id_loss nan |Data 0.023s(0.mot/crowdhuman_dla34 |######### | train: [1][2786/9757]|Tot: 0:17:09 |ETA: 0:44:13 |loss nan |hm_loss nan |wh_loss nan |off_loss nan |id_loss nan |Data 0.023s(0.mot/crowdhuman_dla34 |######### | train: [1][2787/9757]|Tot: 0:17:10 |ETA: 0:44:13 |loss nan |hm_loss nan |wh_loss nan |off_loss nan |id_loss nan |Data 0.023s(0.

[hopesala]

generate odgt file from this guide https://github.com/zengarden/light_head_rcnn/issues/10 and then can train normally

[hopesala]

opps nan again......

mot/crowdhuman_dla34 |############################ | train: [1][3308/3766]|Tot: 0:38:24 |ETA: 0:05:18 |loss 18.8693 |hm_loss 1.1229 |wh_loss 0.2640 |off_loss 0.1886 |id_loss 13.351mot/crowdhuman_dla34 |############################ | train: [1][3309/3766]|Tot: 0:38:25 |ETA: 0:05:17 |loss 18.8684 |hm_loss 1.1228 |wh_loss 0.2640 |off_loss 0.1886 |id_loss 13.351mot/crowdhuman_dla34 |############################ | train: [1][3310/3766]|Tot: 0:38:26 |ETA: 0:05:11 |loss 18.8675 |hm_loss 1.1228 |wh_loss 0.2640 |off_loss 0.1885 |id_loss 13.351mot/crowdhuman_dla34 |############################ | train: [1][3311/3766]|Tot: 0:38:26 |ETA: 0:05:10 |loss nan |hm_loss 1.1227 |wh_loss 0.2639 |off_loss 0.1885 |id_loss nan |Data mot/crowdhuman_dla34 |############################ | train: [1][3312/3766]|Tot: 0:38:27 |ETA: 0:05:09 |loss nan |hm_loss 1.1227 |wh_loss 0.2639 |off_loss 0.1885 |id_loss nan |Data mot/crowdhuman_dla34 |############################ | train: [1][3313/3766]|Tot: 0:38:28 |ETA: 0:05:08 |loss nan |hm_loss 1.1252 |wh_loss nan |off_loss nan |id_loss nan |Data 0.016smot/crowdhuman_dla34 |############################ | train: [1][3314/3766]|Tot: 0:38:28 |ETA: 0:05:04 |loss nan |hm_loss 1.1276 |wh_loss nan |off_loss nan |id_loss nan |Data 0.016smot/crowdhuman_dla34 |############################ | train: [1][3315/3766]|Tot: 0:38:29 |ETA: 0:05:03 |loss nan |hm_loss nan |wh_loss nan |off_loss nan |id_loss nan |Data 0.016s(0.mot/crowdhuman_dla34 |############################ | train: [1][3316/3766]|Tot: 0:38:30 |ETA: 0:04:56 |loss nan |hm_loss nan |wh_loss nan |off_loss nan |id_loss nan |Data 0.016s(0.mot/crowdhuman_dla34 |############################ | train: [1][3317/3766]|Tot: 0:38:30 |ETA: 0:04:55 |loss nan |hm_loss nan |wh_loss nan |off_loss nan |id_loss nan |Data 0.016s(0.mot/crowdhuman_dla34 |############################ | train: [1][3318/3766]|Tot: 0:38:31 |ETA: 0:04:49 |loss nan |hm_loss nan |wh_loss nan |off_loss nan |id_loss nan |Data 0.016s(0.mot/crowdhuman_dla34 |############################ | train: [1][3319/3766]|Tot: 0:38:31 |ETA: 0:04:49 |loss nan |hm_loss nan |wh_loss nan |off_loss nan |id_loss nan |Data 0.016s(0.mot/crowdhuman_dla34 |############################ | train: [1][3320/3766]|Tot: 0:38:32 |ETA: 0:04:42 |loss nan |hm_loss nan |wh_loss nan |off_loss nan |id_loss nan |Data 0.016s(0.

[moon6666]

me too

[xiaotaia]

@hopesala @moon6666 @cgg99 how to solve it?

[zyvonne]

for my custom dataset, the evaluator seems not load the gt.txt actually, the gt_dataset_dict is empty. note use ',' splilt the label info instead of space , and valid label is correct.