Mask R-CNN

[0x00b1]

keras-rcnn should include an implementation of Mask R-CNN.

[JihongJu]

To implement Mask R-CNN we might need four components:

Mask branch: Additional head extending the previous faster R-CNN branches (class and bbox)

RoIAlign Layer: Performs bilinear interpolation whiling pooling instead of

Mask Loss: Performs pixel-to-pixel binary cross entropy loss for the predicted mask and the target mask.

Mask label generator: Yields the corresponding target mask, given cls and bbox coords.

[0x00b1]

I agree. Thankfully, it should be straightforward. I haven’t looked their pooling implementation, but I don’t remember the details in the paper being particularly tricky to be implement. Would you be interested in creating new issues for the items you enumerated? If not, I can do it.

[mayman99]

+1 and I would like to help implementing it.

[0x00b1]

Hi, @MohmadAyman! We’d love your help.

• We still need to write the Mask RCNN model. It should be straightforward and is likely the easiest item to contribute if you’re unfamiliar with the package. I recommend looking at the RCNN implementation (keras_rcnn.models) from the most recent PR from @JihongJu for inspiration:

https://github.com/broadinstitute/keras-rcnn/pull/27/files

• We’ve updated keras_rcnn.layers.pooling.ROI with the appropriate Mask RCNN modifications. However, it’d benefit from additional documentation and unit tests.

• We still need to implement mask loss!

• The ImageSegmentationGenerator should be an easier task. It should look similar to the existing generator except each box includes a "mask" key that has the mask pathname, e.g.

{
    "filename": "example.png",
    "boxes": [
        {
            x1: 0,
            y1: 0,
            x2: 0,
            y2: 0,
            description: "foo",
            mask: "mask.png"
        }
    ],
    "r": 224,
    "c": 224,
    "channels": 3
}

and will yield masks (in addition to an image, bounding boxes, and labels), e.g.

generator = ImageSegementationGenerator()

generator = generator.flow(training)

image, (boxes, labels, masks) = generator.next()

[mayman99]

so if I to start with the easiest, the ImageSegmentationGenerator, would it be the same like the _object_detection.py in the preprocessing folder, and would replace the anchor function with another one with the difference that for every bbox that is returned we would return also the mask, right? e.g.

    for bbox_num, bbox in enumerate(image['boxes']):
        # get the GT box coordinates, and resize to account for image resizing
        gta[bbox_num, 0] = bbox['x1'] * (resized_width / float(width))
        gta[bbox_num, 1] = bbox['x2'] * (resized_width / float(width))
        gta[bbox_num, 2] = bbox['y1'] * (resized_height / float(height))
        gta[bbox_num, 3] = bbox['y2'] * (resized_height / float(height))
        masks[bbox_num] = bbox['mask']

[0x00b1]

Hi, @MohmadAyman! Because we’re moving the anchor generator into the loss function, you don’t need to worry about the anchors. The generator just needs to yield an image, a list of bounding boxes for that image, a list of (instance) masks for that image, and corresponding labels.

[mayman99]

Hi @0x00b1, for testing, which dataset that have the mask annotations would we use?

[jhung0]

Probably MS COCO

On Tue, Jul 4, 2017 at 6:03 AM, Mohmmad Ayman [email protected] wrote:

Hi @0x00b1 https://github.com/0x00b1, for testing, which dataset that have the mask annotations would we use?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/broadinstitute/keras-rcnn/issues/2#issuecomment-312838623, or mute the thread https://github.com/notifications/unsubscribe-auth/AJJbgtX8o4AYOqBhCre5b_OBkb8Og-Yyks5sKg4CgaJpZM4NHgL5 .

[HuangBo-Terraloupe]

Did someone has already created a repository for Mask RCNN, so we all can contribute on it? Really need a repository of keras!

[0x00b1]

Hi, @HuangBo-Terraloupe! We were missing a few pieces. https://github.com/broadinstitute/keras-rcnn/issues/2#issuecomment-311130295

If you're interested in contributing, let me know and I can help steer you in the right direction. 👍

[mayman99]

Sorry I couldn't continue working on it. The COCO data set download speed is too low it would take more than 4 days for me to download it, if anyone know a place other than the official download site or a torrent link that would help.

[HuangBo-Terraloupe]

@0x00b1 I am happy to help, please tell me where we are now, as Jihong Ju mentioned there are 4 steps we need to do. And I have already run the training of Faster-rcnn in keras and Mask Rcnn in tensorflow, the code are available.
I really want to have the implementation of mask rcnn in keras. Maybe I can start with implement the generator.

[HuangBo-Terraloupe]

I want to start with Generator, can someone help me instructions? And for the roi layer in pooling.py, is the layer already ROIAlign? How can I test it?

[mayman99]

@HuangBo-Terraloupe "Hi, @MohmadAyman! Because we’re moving the anchor generator into the loss function, you don’t need to worry about the anchors. The generator just needs to yield an image, a list of bounding boxes for that image, a list of (instance) masks for that image, and corresponding labels."

Here is the generator that I've started working on _image_seg_generator.py .

Although this is not complete, but I hope it gives you an idea to get started.

[manhcuogntin4]

Is there someone that made the training on your own data set ? Before with Faster RCNN, I can use the labelImg tool to create the annotation file for image but I don't know in case of Mask RCNN, what tool I have to use to make the annotations file ? I have to make the mask image my self for the annotation or Mask RCNN will create it automatically ? Thank !