Poor results on Kaggle Carvana challenge

[ddb4ng]

Goal

First of all, thank you very much jakeret et al. for this wonderful project. It is a great piece of work, that is also usable for others.

My goal is to train TensorFlow UNet on the Kaggle Carvana dataset. This is an image segmentation problem with 2 classes: car and background.

Issue

Everything works, but I do not get good results yet (see below).

I am hoping to improve the results by changing hyper parameters or network layers. But which ones should I change and how? I already tried some different hyper parameter values (see below), but without much success.

I looked at the issues that were already posted. Some are related, but no solution yet. Maybe we can bundle our efforts:

• #170
• #168
• #112
• ...?

Sample output

Results generated during cross validation

Below are some results generated during cross validation (left:input, center: ground-truth, right: output - original input data resized to 512x512). These results are not very good, but at least it does something meaningful.

Results generated during testing

Below are some results generated during testing on previously unseen data. These are the results that count, but they are not satisfactory.

Code

/tf_unet/unet.py : Trainer._get_optimizer():

[...]

learning_rate = self.opt_kwargs.pop("learning_rate", 0.4)  # 0.2)
decay_rate = self.opt_kwargs.pop("decay_rate", 0.05)  # 0.95)
momentum = self.opt_kwargs.pop("momentum", 0.5)  # 0.2)

[...]

train.py (not in tf_unet repository):

from tf_unet import unet, util, image_util

data_provider = image_util.ImageDataProvider(
        "training_data_0512-0512_stretched/*",
        data_suffix=".jpg", mask_suffix="_mask.gif")

net = unet.Unet(channels=3, n_class=2, layers=3, features_root=64)
trainer = unet.Trainer(net)
output_path = "output"
path = trainer.train(data_provider, output_path, training_iters=32, epochs=50)  #100)

test.py (not in tf_unet repository):

from tf_unet import unet
import cv2
import numpy as np

net = unet.Unet(channels=3, n_class=2, layers=3, features_root=64)

filelist = ["test1.jpg"]  # , "test2.jpg"]

test_images = np.array([cv2.imread(fname, 1) for fname in filelist])

prediction = net.predict("output/model.cpkt", test_images)

output = prediction[0, :, :, 1] * 1000
# Times 1000 because it is hardly visible otherwise

cv2.imwrite("output.jpg", output)

Discussion

As you can see, the segmentation of car and background is not working well yet. Which hyper parameter values or network layers will lead to success?

Also, why do the cross validation results and the test results look so different? And why do I need to multiply the segmentation by 1000 in order to see the test result? Shouldn't these by binary masks? If you look closely, not all pixels intensities are the same either.

Thank you very much in advance for any help you can provide (anyone).

[jakeret]

Looks like a nice competition!

• For I would check the Tensorboard. The loss should always be decreasing over the epochs. If not something might not be right (with the input).
• If this is ok increase training_iters such that more of your training data is being used.
• Train the net for more epochs
• Try different combinations of layers and filters (e.g. 4 & 32)

The resulting predictions are probability maps. Very small numbers might be a hint that the nework hasn't learned much

[ddb4ng]

Thank you very much for your valuable comments @jakeret . I will let you know how it went :)

[ddb4ng]

Performance got significantly better with your suggestions above. I haven't tried different layers and filters yet though. Also, I got good performance with a different repository, so there is currently no need for me to investigate further.

[jakeret]

Out of curiosity: could you point us to this other repo?

[ddb4ng]

Sorry for my very late reply. I am using and contributing to the semantic segmentation system from TU Eindhoven, MPS group: https://github.com/tuemps/semantic-segmentation . Their system is not ready for public use yet though. I am also using DeepLab (for stuff related to the Carvana dataset): https://github.com/tensorflow/models/tree/master/research/deeplab

[abderhasan]

Hello @ddb4ng,

I'm trying to use test.py as you described, but always getting a black output, why is that?

Are there any changes I have to make after retrieving the saved model?

Thanks.

[ddb4ng]

@abderhasan I didn't get it to work well myself either. Maybe @jakeret can help you. Or you could try this alternative, with which we got really good results on the Kaggle Carvana dataset: https://github.com/petrosgk/Kaggle-Carvana-Image-Masking-Challenge

@jakeret The link above points to the repository that we finally used to get really good results.