Add FCN Model for Segmentation

[suvadityamuk]

Short Description A fresh issue to discuss the implementation of the Fully Convolutional Network for Semantic Segmentation paper Last discussed here in #1099 As mentioned in Roadmap for Q4 2022

Papers Fully Convolutional Network for Semantic Segmentation - 34479 citations, 222 mentions (from arXiv scite_ tool)

Results from paper for task : Semantic Segmentation

Model Backbone	MeanIOU
FCN - AlexNet	39.8
FCN - VGG16	56.0
FCN - GoogLeNet	42.5

Existing Implementations

• Keras-FCN Implementation 1
• Keras-FCN Implementation 2

Other Information I think we should have a discussion on the API for the model. Backbones can vary significantly for this model although an almost-standardized method uses the VGG16 and VGG19 models.

@IMvision12 has already begun working on a draft implementation for the VGG16 backbone. We will be collaborating and co-authoring this model.

We can keep the ResNet-50, VGG16 and VGG19 models as supported backbones for the model with their model configs present for use.

Would we want to allow custom backbones? If so, how would we do that?

• Extract convolutional layers/blocks from an existing tf.keras.Model instance passed as a parameter, which would presumably be a plain CNN.
• Custom, list-based parameters such as depth_kernel_sizes, depth_num_filters, and depth_padding_modes which would be then used to generate the backend in an iterative fashion.
• Accept a list of tf.keras.layers.Conv2D / tf.keras.Sequential which would themselves be the backbones' implementation

@bhack @ianstenbit @LukeWood @tanzhenyu

[DavidLandup0]

Would we want to allow custom backbones? If so, how would we do that?

That's planned for object detection, and IMO, should be for semantic segmentation too if not already. In #1128, the default backbone is a ResNet101, but the user can supply a keras_cv.models.Model and define the layer names to be used for low-level and high-level feature maps.

For example:

backbone = keras_cv.models...

model = segmentation.DeepLabV3Plus(
            classes=11,
            include_rescaling=True,
            backbone=backbone,
            feature_layers=("v2_stack_1_block4_1_relu", "v2_stack_3_block2_2_relu"),
            input_shape=(256, 256, 3),
        )

[suvadityamuk]

Would we want to allow custom backbones? If so, how would we do that?

That's planned for object detection, and IMO, should be for semantic segmentation too if not already. In #1128, the default backbone is a ResNet101, but the user can supply a keras_cv.models.Model and define the layer names to be used for low-level and high-level feature maps.

For example:

backbone = keras_cv.models...

model = segmentation.DeepLabV3Plus(
            classes=11,
            include_rescaling=True,
            backbone=backbone,
            feature_layers=("v2_stack_1_block4_1_relu", "v2_stack_3_block2_2_relu"),
            input_shape=(256, 256, 3),
        )

Alright, makes sense. Are we sure we want to go ahead with this?

[tanzhenyu]

Would we want to allow custom backbones? If so, how would we do that?

That's planned for object detection, and IMO, should be for semantic segmentation too if not already. In #1128, the default backbone is a ResNet101, but the user can supply a keras_cv.models.Model and define the layer names to be used for low-level and high-level feature maps. For example:

backbone = keras_cv.models...

model = segmentation.DeepLabV3Plus(
            classes=11,
            include_rescaling=True,
            backbone=backbone,
            feature_layers=("v2_stack_1_block4_1_relu", "v2_stack_3_block2_2_relu"),
            input_shape=(256, 256, 3),
        )

Alright, makes sense. Are we sure we want to go ahead with this?

We would like all our meta architectures (detection architectures, segmentation architectures, and any future architectures) to take custom backbones

[suvadityamuk]

We would like all our meta architectures (detection architectures, segmentation architectures, and any future architectures) to take custom backbones

Is there any specification on how we should accept the backbone? Or should we use @DavidLandup0 's method?

[tanzhenyu]

We would like all our meta architectures (detection architectures, segmentation architectures, and any future architectures) to take custom backbones

Is there any specification on how we should accept the backbone? Or should we use @DavidLandup0 's method?

You can follow the examples under keras_cv/models/object_detection and keras_cv/models/segmentation

[suvadityamuk]

Alright, makes sense. Thank you!

[suvadityamuk]

So our current implementation is looking at adding the VGG16 and VGG19 models as backbones. Should we look at any other model backbones?

[tanzhenyu]

So our current implementation is looking at adding the VGG16 and VGG19 models as backbones. Should we look at any other model backbones?

given this is modular design, you can start with whatever backbone that is available. We have been using keras.applications backbones but is migrating to keras_cv.models backbones

[suvadityamuk]

Makes sense. The VGG16 backbone is now available as in keras_cv.models as part of #1164

For now, I'll use both keras.applications backbones, but we can look for a refactor in the future.

Sending a PR your way with a preliminary API implementation in a while. Training script and tests still remain though. Will work on it once the API is confirmed.