High overall accuracy, low class accuracy and bad predictions

[eusebioaguilera]

trafficstars

Hi! First of all I like to thank you about this great project and good code. I am using your project to try to segment some medical images with two classes (Background and Roots). To do this I have used mainly unet networks (unet, unet_mini, vgg_unet, resnet50_unet, mobilenet_unet, etc). I have also try some other models (segnet, pspnet, etc). However the results that I have obtained are the same, high overall accuracy but low accuracy for the roots segmentation class. Nevertheless, when I visualize the dataset, the part that I try to segment is not a small one:

I use the last version of the code in this repo. I have used one channel labelled images (and also three channel images), PNG format for both labelled images and images. I don't know what's happening. Any advice?

Thanks a lot, in advance.

[divamgupta]

Try fine-tunning from a pre-trained PSPNet model. Accuracy is not a good metric for segmentation. Use IoU instead. Use the evaluate function provided.

[eusebioaguilera]

I have used the evaluate method and I've obtained this values:

{'frequency_weighted_IU': 0.938984987607353, 'mean_IU': 0.3230076304757516, 'class_wise_IU': array([9.69008864e-01, 1.40271538e-05])}

Ok. I will try the PSPNet pretrained models.

[eusebioaguilera]

Hi, Hi, first of all, thank you for the response.

I have follow the steps described in the fine-tuning from a pre-trained model and train using my dataset. Using the evaluate function provided I obtain the following results:

{'frequency_weighted_IU': 0.9390266035920913, 'mean_IU': 0.48451692529572465, 'class_wise_IU': array([0.96903385, 0. ])}

So far, the model seems that is not learning how to segment the class that I need. Any suggestion?

[divamgupta]

What is the size of your dataset? Try using data augmentation. For a sanity check, what is the IoU on the train images?

[eusebioaguilera]

Hi, thank you for the response.

My dataset has 1600 images for training and 360 for validation. I have been training the model for 50 epochs and I have obtained a IoU over the roots segmentation class of 0.0226 in the validation images. I have used the evaluation method over the train images and a get the values:

{'frequency_weighted_IU': 0.9402407781527442, 'mean_IU': 0.4848043284815004, 'class_wise_IU': array([0.96960866, 0. ])}

The problem is the size of the dataset?

Thank you.

[divamgupta]

Could you share a few samples of your prepared dataset? And the exact code/commands you are using to train the model. I can look into it.

[eusebioaguilera]

Hi, first of all, thank you for the response.

I have reviewed the generation code and I have found an error when the dataset was generated. So the net obtained a dataset where some images and segmentation were mislabelled. After generated the dataset again I have tests with different models and I have obtained the best results with the resnet50_unet model. Training this model for 50 epochs I have obtained an IOU of 0.81281024 for the segmentation class. I have try to train the model more, but the results seems not to improve.

The code that I have used to train and evaluate is the following train_keras_seg.py.txt and some examples of the dataset are dataset.zip

The results that I have obtained are good, however is there a way of improve them?

Thank you.

[mayi666]

Hi, Hi, first of all, thank you for the response.

I have follow the steps described in the fine-tuning from a pre-trained model and train using my dataset. Using the evaluate function provided I obtain the following results:

{'frequency_weighted_IU': 0.9390266035920913, 'mean_IU': 0.48451692529572465, 'class_wise_IU': array([0.96903385, 0. ])}

So far, the model seems that is not learning how to segment the class that I need. Any suggestion?

Hello,How did you get to frequency_weighted_IU,mean_IU,class_wise_IU? I use the author’s terminal command line, but the terminal only displays the loaded weights. No values are printed, and I don’t know where to save them. Thanks!

[eusebioaguilera]

Hi, Hi, first of all, thank you for the response. I have follow the steps described in the fine-tuning from a pre-trained model and train using my dataset. Using the evaluate function provided I obtain the following results: {'frequency_weighted_IU': 0.9390266035920913, 'mean_IU': 0.48451692529572465, 'class_wise_IU': array([0.96903385, 0. ])} So far, the model seems that is not learning how to segment the class that I need. Any suggestion?

Hello,How did you get to frequency_weighted_IU,mean_IU,class_wise_IU? I use the author’s terminal command line, but the terminal only displays the loaded weights. No values are printed, and I don’t know where to save them. Thanks!

If you see the code in the previous message, you will see that I have used the api provided by the keras_segmentation framework. The evaluation code is as simply as:

print(model.evaluate_segmentation( inp_images_dir=str_val_dir , annotations_dir=str_val_ann_dir, checkpoints_path=str_checkpoint_dir ) )

This gives you the values that you want.

[mayi666]

Hi, Hi, first of all, thank you for the response. I have follow the steps described in the fine-tuning from a pre-trained model and train using my dataset. Using the evaluate function provided I obtain the following results: {'frequency_weighted_IU': 0.9390266035920913, 'mean_IU': 0.48451692529572465, 'class_wise_IU': array([0.96903385, 0. ])} So far, the model seems that is not learning how to segment the class that I need. Any suggestion?

Hello,How did you get to frequency_weighted_IU,mean_IU,class_wise_IU? I use the author’s terminal command line, but the terminal only displays the loaded weights. No values are printed, and I don’t know where to save them. Thanks!

If you see the code in the previous message, you will see that I have used the api provided by the keras_segmentation framework. The evaluation code is as simply as:

print(model.evaluate_segmentation( inp_images_dir=str_val_dir , annotations_dir=str_val_ann_dir, checkpoints_path=str_checkpoint_dir ) )

This gives you the values that you want.

Thank you very much for your reply, the problem is solved.