Sudden increment in loss values during training

[purvang3]

[xiaohu2015]

@purvang3 do you test your dataset by other detection models?

[purvang3]

Thank you for the reply. It was dataset only. Could you point out to the document and responsible script, which periodically evaluate saved checkpoint on validation data?

[xiaohu2015]

@purvang3 you can reference detectron2 document. For the checkpoint, you can set https://github.com/facebookresearch/detectron2/blob/main/detectron2/config/defaults.py#L541 for eval, you can set https://github.com/facebookresearch/detectron2/blob/main/detectron2/config/defaults.py#L592

[purvang3]

@xiaohu2015 Thank you for the reply and pointing out to the doc regarding eval. I am able to run periodic eval but coco metric validation result show either garbage values or not improving mAP but I do see decrement in training loss. Below I have attached image. Do you have any suggestion? let me know if you need any additional information.

[xiaohu2015]

check your dataset, use https://github.com/facebookresearch/detectron2/blob/main/tools/visualize_data.py to show dataloader to check if your data is good.