Prediction is different when the model is loaded

[yashkan27]

I have trained a vgg_unet model for segmenting buildings,roads,water bodies and vegetation. The accuracy of the model is 94%. When I get the prediction from the model it is good. But when I saved the model (through .save_weights and by model_from_checkpoint_path method the results is very different). I think I am missing something but the prediction when the model is loaded outputs the labels whereas when the model is in memory it masks the label with original image. Thanks

[yashkan27]

Update: I was missing the overlay_img option but here is the difference between the GPU prediction and CPU prediction

CPU Prediction:

GPU Prediction:

[divamgupta]

Could you share the code/colab notebook by which this can be reproduced?

[yashkan27]

I am using the same colab notebook that you mention in the README. The only difference is this is my custom data and I label 5 class in every image(Also check the it with np.unique()). I am training this on the local GPU. When I get prediction from the model using the model_from_checkpoint_path() from the separate terminal everything is good than I transfer the same python script, weights along with the vgg_unet_1_config.json to my PC. Ideally I should be getting the same results but this time it is all coming to same class. Is there anything related to the GPU config? Thanks.

[kim0522ms]

@divamgupta @yashkan27 Same issue on here. I got different prediction result when I load model trained on GPU to CPU-machine. Did you find any solution about this problem?

[divamgupta]

Maybe some library version is different?

On Tue, Apr 6, 2021 at 3:23 AM MS Kim @.***> wrote:

@yashkan27 https://github.com/yashkan27 Same issue on here. I get different prediction result when I load model trained on GPU to CPU-machine. Did you find any solution about this problem?

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/divamgupta/image-segmentation-keras/issues/244#issuecomment-813890979, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAONR5PS4IULBBNLVMB34DLTHKZGTANCNFSM4SF6X5MA .

[yashkan27]

@kim0522ms As far as I know, the situation arises due to some calculation method that is slightly different in GPU and CPU. I get 0 as the intermediate result in the training for one and Nan for the other even if I keep all the things constant(All python libraries except the CUDA).

Hope this helps in some way. @divamgupta

[rola93]

could it be related to #235 ? I'm having similar issue: train on GPU, save the model, load back again and then results are not as good as expected (I'm having mIOU ~0.8). Then, when I load the model to GPU or CPU results are very very bad.

My guess is that both are related and the problem is when doing model.save; I may be missing something...

[sachinkmohan]

Same here, after model.save and loading the model, my predictions are also bad. Any help is appreciated.