Question about validation

[CeciliaPYY]

Hello, first thanks for your pytorch version of deeplab v3+, I try it at last days of 2018, and I got a perfect model whose performance is great both on train and validation datasets, which means their loss are steadily decreasing at my first try. However, when I give it a second try, train loss of the new model still decreased but the validation loss went up and down sometimes.

Below, is the first try performance. train

validation

Below, is the second try performance. train

validation

ps, the dataset of two try is definitely the same!! So, do you have any idea of this?

[sonack]

did you used the syncd bn option？

[CeciliaPYY]

did you used the syncd bn option？

I don't think I used it cause I just have one gpu to train it.

[Ellsionjeep]

@CeciliaPYY How to get kinds of figures ,can you give me a copy? thanks