Does VGG Segmentations (polygons) can be augmented ?

[nayzen]

Hi @aleju, and thanks for this amazing library !

I use Mask R-CNN from Matterport Github, and recently with your library because I need more images. The "problem" is that my annotations are polygons made with VGG Image Annotator tool and I want to also augment these annotations.

My coordinates are in a .Json file like this:

{"ADAPTATEUR_REMPLISSEUR_4134C074_FC1_0.jpg2101333":{"fileref":"", "size":2101333, "filename":"ADAPTATEUR_REMPLISSEUR_4134C074_FC1_0.jpg", "base64_img_data":"", "file_attributes":{}, "regions":{"0":{ "shape_attributes":{ "name": "polygon", "all_points_x":[1040,1042,1045,1032,1032,1042,1057,1081,1136,1159,1176,1185,1188,1188,1197,1358,1354,1336,1317,1319,1310,1049,1040], "all_points_y":[1108,1023,1008,925,899,875,855,838,804,799,807,819,830,909,922,994,1020,1049,1061,1110,1112,1120,1108]}, "region_attributes":{"class_id":"1"}}}}

For the moment I augmented my databse without any geometric transformation, but if I want to do rotations, or translations.. what will I need to do to transform/ load my annotations ?

Best regards, Antoine

[aleju]

There is a Polygon class in the library, but it cannot be augmented yet. Your best options are:

1. Staying with your method of only applying non-geometric augmentations.
2. Converting the polygons to imgaug.Keypoint objects (and wrapping them in imgaug.KeypointsOnImage objects) and augmenting these via augment_keypoints(). This might cause the polygons to be non-concave (e.g. overlapping parts) for PiecewiseAffine or ElasticTransformation. But Affine and PerspectiveTransform should not cause such issues.
3. Converting the polygons to imgaug.HeatmapsOnImage or imgaug.SegmentationMapOnImage objects and augmenting these via augment_heatmaps() and augment_segmentation_maps(). As both of these cases will augment image-like arrays, they are going to be slower than keypoint augmentation. You might also have to re-convert the arrays to the corner points of polygons, e.g. via some kind of contour finding and then simplification.

For (2) and (3), the readthedocs documentation has chapters that explain how to handle these augmentation types.

[nayzen]

Thanks for your response Aleju !

I'm going to try this !

[nayzen]

Hi Aleju !

I want to do your second option (convert the polygons to imgaug.Keypoint objects). I tried it but manually and it works fine (see results below)

But when you say "converting the polygons to imgaug.keypoints", you mean that I need to find a way to extract my coordinates from my .json file ? I have 2800 coordinates on the same line, in the same file. Maybe there is a "converter" to make this task easier ?

Thank you again for your help, Regards, Antoine

[aleju]

You can parse json files fairly easily with the json module. Something in the direction of

import json
with open("annotations.json", "r") as f:
    content = json.load(f)
    for polygon in polygons:  # there are probably multiple polygons in the file? so this "for" must be somehow derived from the file content
        xx = content["some"]["parent"]["keys"]["all_points_x"]  # "..." needs to be filled with some keys
        yy = content["some"]["parent"]["keys"]["all_points_y"]
        keypoints = [ia.Keypoint(x=x, y=y) for x, y in zip(xx, yy)]
        # do something with the keypoints

[nayzen]

Thanks a lot @aleju for your help :-)

[Ajaykumarrachuri]

Does VGG segmentations-(bounding boxes ) can be augmented?

[pushkalkatara]

@nayzen Did you implement this with augumenter on the MaskRCNN repository?

[pushkalkatara]

just read your comment https://github.com/matterport/Mask_RCNN/issues/1015 @nayzen . Automatically takes the key points and augment it, the library is designed in that way. Thanks.

[BennyGinger]

Thanks a lot @aleju for your help :-)

Hello @nayzen ,

It's a long shot, but how do you implement the keypoints into the model? Especially, how do you convert the keypoints_polygons into mask. Any chance you could share your code with us? Merci!

[3goto]

You can parse json files fairly easily with the json module. Something in the direction of

import json
with open("annotations.json", "r") as f:
    content = json.load(f)
    for polygon in polygons:  # there are probably multiple polygons in the file? so this "for" must be somehow derived from the file content
        xx = content["some"]["parent"]["keys"]["all_points_x"]  # "..." needs to be filled with some keys
        yy = content["some"]["parent"]["keys"]["all_points_y"]
        keypoints = [ia.Keypoint(x=x, y=y) for x, y in zip(xx, yy)]
        # do something with the keypoints

How should I change it if I rewrite it like this questioner?

[FloBr49]

Hi everyone,

i have to prepare a project for college and i'm tryin to do it with matterport Mask RCNN. My problem is now that i have images with lots of annotations (around 25 annotations with 8 different classes). To collect enough annotated data is too time consuming. So my question is: How do i use the image augmentation, especially geometric transformation, to enlarge my training dataset and automatically transform my annotations?

You can find an example JSON file I want to use attached.

Thank you in advance.

TEST_JSON_FB.zip