Loss problem

[HanSolo09]

Hi, everyone, I have a problem when I try to retrain the model with the NYUv2 datset, including around 1k imgs with 14 classes. (loss='categorical_crossentropy') Regardless of whether I freeze the weights, when the training runs to the second epoch, the loss suddenly becomes minimal and does not change. (The first epoch looks normal, the loss is falling, but the acc is oscillating) Does anyone know how to solve this problem?

[mazenmel]

Same problem with NYUv2

[trungpham2606]

@bonlime so with this new version , are we able to either retrain it with your provided pre-trained weights or fine-tune it ?

[wenze-shui]

hi, I have the same problem. Have you solved it?

[Fuyaoyao]

Hi,I have the same problem when I train network with 2 GPUs. Have you met it?

[wenze-shui]

Hi,I have the same problem when I train network with 2 GPUs. Have you met it?

I added an activation layer at the end of the network, and now loss looks normal.

[Fuyaoyao]

Hi,I have the same problem when I train network with 2 GPUs. Have you met it?

I added an activation layer at the end of the network, and now loss looks normal.

We can see that it exists activation layer in Line 454 in model.py .Could you please tell me which activation layer you add in ？

[wenze-shui]

嗨，我训练带有2个GPU的网络时遇到同样的问题。你见过吗？

我在网络末端添加了一个激活层，现在丢失看起来很正常。

我们可以在model.py的454行中看到它存在激活层。能否请您告诉我添加了哪个激活层？

Sorry, my tensorflow version is 1.13. I didn't use the latest code. I may not be able to help you.

[Fuyaoyao]

嗨，我训练带有2个GPU的网络时遇到同样的问题。你见过吗？

我在网络末端添加了一个激活层，现在丢失看起来很正常。

我们可以在model.py的454行中看到它存在激活层。能否请您告诉我添加了哪个激活层？

Sorry, my tensorflow version is 1.13. I didn't use the latest code. I may not be able to help you.

OK，thank you very much for your reply.

[xiaokeaiyayaya]

hi,Ialso have the same problem,if I train on voc2012 (do not load pretrained paras),the result is wrong?so anyone solved? please please please please