Trying to access the location or coordinates of predicted masks Mask-RCNN

[DanishMalik333]

We are working on a project, vision controlled robot for pick and place partial occluded objects. We are using mask RCNN architecture. Until now, we are successful in producing masks, bounding boxes and getting the location of bboxes. Now to handle occlusion effectively we need the coordinates of the masks, so that we can identify overlapped area. But it is not working. Here is a code snap and result it producing.

import skimage
import matplotlib.pyplot as plt

real_test_dir = '/content/drive/MyDrive/MaskRCNN_occ/dataset/updated data 2/predict'
image_paths = []
for filename in os.listdir(real_test_dir):
    if os.path.splitext(filename)[1].lower() in ['.png', '.jpg', '.jpeg']:
        image_paths.append(os.path.join(real_test_dir, filename))

for image_path in image_paths:
    img = skimage.io.imread(image_path)
    
    if img.ndim != 3:
      image = skimage.color.gray2rgb(img)
    if img.shape[-1] == 4:
      image = img[..., :3]

    print(image_path)
    img_arr = np.array(image)
    results = model.detect([img_arr], verbose=1)
    r = results[0]
    print(r['masks'])

And it is producing these results,

/content/drive/MyDrive/MaskRCNN_occ/dataset/updated data 2/predict/Image 3.png
Processing 1 images
image                    shape: (256, 256, 3)         min:    5.00000  max:  255.00000  uint8
molded_images            shape: (1, 512, 512, 3)      min: -117.70000  max:  151.10000  float64
image_metas              shape: (1, 18)               min:    0.00000  max:  512.00000  float64
anchors                  shape: (1, 65472, 4)         min:   -0.17712  max:    1.05188  float32
[[[False False False False False]
  [False False False False False]
  [False False False False False]
  ...
  [False False False False False]
  [False False False False False]
  [False False False False False]]]

[sohinimallick]

Have the same problem. I need the pixel co-ordinates/location of the predictions so that I can write it to text file and convert to GEOJSON format for gis. Anyluck?

[DanishMalik333]

Not yet because it is working on every single pixel and classifying it individually. Mask is not being stored in form of any polygon whose coordinates can be found.

[sohinimallick]

Not yet because it is working on every single pixel and classifying it individually. Mask is not being stored in form of any polygon whose coordinates can be found.

import numpy as np from imantics import Polygons, Mask import numpy as np from imantics import Polygons, Mask

array = np.ones((100, 100)) //can be any array

polygons = Mask(array).polygons()

print(polygons.points) print(polygons.segmentation)

I found this script that someone had written to convert the masks, but I am not sure if this is what is required?

[DanishMalik333]

Array here is not specified because r['masks'] gives us array in terms of true and false, which is not a valid input to create polygons.

[st162053]

Did you manage to find a solution for this isssue?

[Ankit11411]

I have found written code in display_instances() function in visualize.py file. I have made modification on this code and made separate function which is below:

from skimage.measure import find_contours
import numpy as np


def save_co_ordinates(image, boxes, masks, class_ids, class_names):
    image = image.split("/")[-1]
    image_data = []

    for i in range(boxes.shape[0]):

        class_id = class_ids[i]
        label = class_names[class_id]

        mask = masks[:, :, i]
        padded_mask = np.zeros(
            (mask.shape[0] + 2, mask.shape[1] + 2), dtype=np.uint8)
        padded_mask[1:-1, 1:-1] = mask
        contours = find_contours(padded_mask, 0.5)
        for verts in contours:
            verts = np.fliplr(verts) - 1
            list_co_ordinates = np.moveaxis(verts, 1, 0).tolist()

            region = {"shape_attributes": {"all_points_x": list_co_ordinates[0],
                                           "all_points_y": list_co_ordinates[1]},
                      "region_attributes": {"name": {label: True}}}
            image_data.append(region)
    data = {"filename": image, "regions": image_data}
    return data

detection = {1:  'apple', 2: 'banana', 3:'orange'}   

save_co_ordinates("C:/path/apple.jpg", r1['rois'], r1['masks'], r1['class_ids'], detection)

[w-oceanfish]

I have found written code in display_instances() function in visualize.py file. I have made modification on this code and made separate function which is below:

from skimage.measure import find_contours
import numpy as np


def save_co_ordinates(image, boxes, masks, class_ids, class_names):
    image = image.split("/")[-1]
    image_data = []

    for i in range(boxes.shape[0]):

        class_id = class_ids[i]
        label = class_names[class_id]

        mask = masks[:, :, i]
        padded_mask = np.zeros(
            (mask.shape[0] + 2, mask.shape[1] + 2), dtype=np.uint8)
        padded_mask[1:-1, 1:-1] = mask
        contours = find_contours(padded_mask, 0.5)
        for verts in contours:
            verts = np.fliplr(verts) - 1
            list_co_ordinates = np.moveaxis(verts, 1, 0).tolist()

            region = {"shape_attributes": {"all_points_x": list_co_ordinates[0],
                                           "all_points_y": list_co_ordinates[1]},
                      "region_attributes": {"name": {label: True}}}
            image_data.append(region)
    data = {"filename": image, "regions": image_data}
    return data

save_co_ordinates("C:/path/apple.jpg", r1['rois'], r1['masks'], r1['class_ids'], get_detections())

Hi, can you explain what is get_detections()? Where can I access to this function?

[NikeeShrestha]

Hi Ankit,

Can you explain where did you add this function?

[Ankit11411]

Hi Ankit,

Can you explain where did you add this function?

I have updated my answer. you would have assigned integer to the detection class while training. that is what I was calling with that function