SSD-VGG: Loss nan

[hoangcuongbk80]

Hi,

I trained the SSD for my own dataset using VGG and get loss nan as below:

I trained my own data on Jetson Xavier. It's ok if training using MobileNet. Any suggestion to make it work with VGG?

[olibartfast]

I had your same error, I completed the training using a lower learning rate i.e --lr 0.0001

[hoangcuongbk80]

@francescooliva Thank you for sharing your experience. I'll try it.

[olibartfast]

Also I can see from your screenshot it looks like you are using mobilenet training parameters, or am I wrong? check in the Readme the Vgg model training instructions: python train_ssd.py --datasets ~/data/VOC0712/VOC2007/ ~/data/VOC0712/VOC2012/ --validation_dataset ~/data/VOC0712/test/VOC2007/ --net vgg16-ssd --base_net models/vgg16_reducedfc.pth --batch_size 24 --num_epochs 200 --scheduler "multi-step” —-milestones “120,160”