Loss starting at 0.2

[apappu97]

Hi @www0wwwjs1! I had a quick question -- I just set up this code and am running it, and on both MNIST and smallNORB the loss starts at <1 (usually around 0.3) and flatlines during training. I'm confused as to why the loss starts so low and doesn't change. When testing on smallNORB, the accuracy is quite low, around 0.45. Please let me know what you think, thank you!

[apappu97]

I'm also curious what the optimal LR for smallNORB was -- got high accuracy for MNIST after ~8k iterations, but the loss is slowly increasing on smallNORB, so wondering what the optimal LR is

[lithostark]

Hi! Have you fixed the problem? I'm running with mnist and the loss starts around 0.3 and jumps to 7.2 after 11000 iterations and stuck at 7.3 when finished at 55000. The testing accuracy is around 0.1 all the time. Really appreciate your help!