Multi-label semantic segmentation evaluation

[ntelo007]

Hi there,

I would like to get some advice from you guys regarding on how a prediction should be properly evaluated. I created a U-Net model with resnet34 as backbone to semantically segment images to 37 different classes. The problem is that in every image only a maximum of 7 classes might appear and in most cases only 2-3 classes appear per image. The result is that the final loss value of the test set is very low because the general comparison with all the 37 classes is successful. E.g. if 20 classes don't appear in an image and the prediction image also doesn't identify any pixels belonging to those classes, the accuracy is 100% and loss 0.

Question: Can you tell me how to get per class accuracy? I think it is more reliable than the overall accuracy or IoU score that this metric gives me.

[JordanMakesMaps]

So you might be better off looking at something like precision and recall instead of just accuracy, although IoU is a fairly good metric for semantic segmentation.

After training, you could feed into the model all of your images, obtain the corresponding predictions for each one, and compare them with their original dense labels using Mean Accuracy found here.

[ntelo007]

I will reverse the one hot encoding and then try to evaluate my images because if I don't, I receive really high accuracy without this being the truth. I will try to use the script that you provided with your link.

[SabraHashemi]

hi i have 3 folders: ground truth, input images , and result(multilable segmented image ) i want to have a evaluation on them, can you provide any code for this? @JordanMakesMaps @ntelo007 @gazay @zcoder @habi

[JordanMakesMaps]

@sabrabano0 if you want to quickly get an evaluation without using Keras, you can use this library. Also, you could just use scikit-learn's metric functions.