Big loss after hours of training

[maxritter]

Thanks for your great implementation of Tensorflow LPRNet!

Unfortunately, training on my own synthetic german license plate data does not give me good results so far. I have 10k images and trained it for three hours on a Nvidia Tesla K80, but the loss is still at around 13.0 (step 8000).

From the checkpoint you used in your "Test Single Image" section, you had a loss of 0.215 at step 5000. Is there anything I am missing here or do you have a tip what could be the reason? Maybe you could also shared your trained model, that would be awesome :)

[bluesy7585]

Hi

Can you provide some examples of your training images?

things I can think of for now:

• did you change CHARS in LPRnet.py to your character set?
• are all labels correct? I saw some german plates has special characters (like ä ?). you will need additional char in CHARS and label for them. e.g lower case 'a'
• are there double line number plates in your training image? are there stacked chars in image? LPRnet does not work for double line
• does your images cover a large range of aspect ratio? for example from 1:1 to 7:1, not sure it works for such a different ratio.
• does it keep at high loss for a long time? maybe it stuck in local minima. try to restart training from earlier checkpoints
• during my training, I notice sometimes loss drops dramatically in one epoch. like 28 to 3 in 1000 steps. maybe try a few more steps