I can't get the mAP, mAR, F1_score using utils.compute_ap(..)

[alcarazolabs]

Hello community from Mask_RCNN I'm trying to get this indicators ( mAP, mAR and F1_score) usign utils.compute_ap(..) here is my code:

from mrcnn.model import load_image_gt
from mrcnn.model import mold_image
from mrcnn.utils import compute_ap, compute_recall
from numpy import expand_dims
from mrcnn import utils
def evaluate_model(dataset, model, cfg):
  APs = list(); 
  F1_scores = list(); 
  for image_id in dataset.image_ids:
      #image, image_meta, gt_class_id, gt_bbox, gt_mask = load_image_gt(dataset, cfg, image_id, use_mini_mask=False)
      image, image_meta, gt_class_id, gt_bbox, gt_mask = load_image_gt(dataset, cfg, image_id)
      scaled_image = mold_image(image, cfg)
      sample = expand_dims(scaled_image, 0)
      yhat = model.detect(sample, verbose=0)
      r = yhat[0]
      AP, precisions, recalls, overlaps = utils.compute_ap(gt_bbox, gt_class_id, gt_mask, r["rois"], r["class_ids"], r["scores"], r['masks'])
      AR, positive_ids = compute_recall(r["rois"], gt_bbox, iou=0.2)
      ARs.append(AR)
      F1_scores.append((2* (mean(precisions) * mean(recalls)))/(mean(precisions) + mean(recalls)))
      APs.append(AP)

  mAP = mean(APs)
  mAR = mean(ARs)
  return mAP, mAR, F1_scores

Then I use the function:

mAP, mAR, F1_score = evaluate_model(dataset_val, model, inference_config)
print("mAP: %.3f" % mAP)
print("mAR: %.3f" % mAR)
print("first way calculate f1-score: ", F1_score)

F1_score_2 = (2 * mAP * mAR)/(mAP + mAR)
print('second way calculate f1-score_2: ', F1_score_2)

I get this code from here: https://github.com/matterport/Mask_RCNN/issues/2165 but when I run this code I'm getting some errors when execute this line of code: mAP, mAR, F1_score = evaluate_model(dataset_val, model, inference_config)

ValueError                                Traceback (most recent call last)
<ipython-input-52-fbdd8a8acc10> in <module>()
----> 1 mAP, mAR, F1_score = evaluate_model(dataset_val, model, inference_config)
      2 
      3 """
      4 print("mAP: %.3f" % mAP)
      5 print("mAR: %.3f" % mAR)

3 frames
/content/matterport/mrcnn/utils.py in compute_overlaps_masks(masks1, masks2)
    113 
    114     # intersections and union
--> 115     intersections = np.dot(masks1.T, masks2)
    116     union = area1[:, None] + area2[None, :] - intersections
    117     overlaps = intersections / union

<__array_function__ internals> in dot(*args, **kwargs)

ValueError: shapes (5,262144) and (3136,5) not aligned: 262144 (dim 1) != 3136 (dim 0)

I'm loading my validation dataset of this way:

# Validation dataset
dataset_val = CustomConfig()
dataset_val.load_cars("/content/MaskRCNN_Video/images", "val")
dataset_val.prepare()

My validation images has a file called via_region_data.json which is the ground truth of these images.. I have trained my model without problems but now when I'm trying to get the mAP, mAR, F1_score with the code of above I'm getting this error:

ValueError: shapes (5,262144) and (3136,5) not aligned: 262144 (dim 1) != 3136 (dim 0)

The same happen when I try to run this code to get the mAP

why can happens it? what should I do? any recommendation? thanks in advance.

[LSnyd]

Hi @alcarazolabs, you have to set USE_MINI_MASK to False in your configs

[alcarazolabs]

Hi @LSnyd I tried that but it didn't work. What I'm doing is train the MaskRCNN using the implementation which works with tensorflow 2.x and after to can get this indicators I am using tensorflow 1.3.0 as the original MaskRCNN for it I'm using anaconda jupyter notebook. The training was done using colab with tensorflow 2.4.1. After it I'm just download the weights and using tensorflow 1.3.0 with jupyter notebook to test the network with those indicators in local mode.

I can't use mAP, mAR, F1_score = evaluate_model(dataset_val, model, inference_config) using tensorflow 2.4.1 or 2.x but I can train the Mask RCNN more faster than mask rcnn using tensorflow 1.3.0. I don't know why it goes very slow in google colab to fix it, I'm using the implementation of Mask RCNN with tensorflow 2.x and it go fast.

[email protected] Best regards.

[sohinimallick]

@alcarazolabs just to be clear, you are training the model in tensorflow 2.x but running the inference in the 1.3.0? And the error dissapears? Do you have any idea of why this is so?

[alcarazolabs]

@sohinimallick yes I did that. The changes are in the model.py file of the Mask R-CNN. For tensorflow 2.x is one version and for tensorflow 1.3.0 is another version. Try and check.

[sohinimallick]

@sohinimallick yes I did that. The changes are in the model.py file of the Mask R-CNN. For tensorflow 2.x is one version and for tensorflow 1.3.0 is another version. Try and check.

@alcarazolabs My model.py implementation is already the tensorflow 2.0 version. I am still getting the error in visualise.display_differences()

[alcarazolabs]

@sohinimallick download your weights and test it using tensorflow 1.3.0

[geomaticsbetul]

hello @alcarazolabs Did you solve the problem? I have 5 classes in my dataset. How many classes did you have? How can I calculate the F1 score? please help, thanks.

[alcarazolabs]

@geomaticsbetul yes I solved it. Just download MaskRcnn folder of 1.3.0 version.. the matterport report has it. I just trained my model using one class with the implementation of MaskRCNN with tensorflow 2.x I just downloaded my weigths and I computed the mAp, mAr and F1_score using Tensorflow 1.30.

#Función para evaluar el modelo con las imágenes de validacion 10%
from mrcnn.model import load_image_gt
from mrcnn.model import mold_image
from mrcnn.utils import compute_ap, compute_recall
from mrcnn import utils

from numpy import zeros
from numpy import asarray
from numpy import expand_dims
from numpy import mean

def evaluate_model(dataset, model, cfg):
    APs = list(); 
    ARs = list();
    F1_scores = list(); 
    for image_id in dataset.image_ids:
        image, image_meta, gt_class_id, gt_bbox, gt_mask = load_image_gt(dataset, cfg, image_id, use_mini_mask=False)
        #image, image_meta, gt_class_id, gt_bbox, gt_mask = load_image_gt(dataset, cfg, image_id)
        scaled_image = mold_image(image, cfg)
        sample = expand_dims(scaled_image, 0)
        yhat = model.detect(sample, verbose=0)
        r = yhat[0]
        AP, precisions, recalls, overlaps = utils.compute_ap(gt_bbox, gt_class_id, gt_mask, r["rois"], r["class_ids"], r["scores"], r['masks'])
        AR, positive_ids = compute_recall(r["rois"], gt_bbox, iou=0.2)
        ARs.append(AR)
        F1_scores.append((2* (mean(precisions) * mean(recalls)))/(mean(precisions) + mean(recalls)))
        APs.append(AP)

    mAP = mean(APs)
    mAR = mean(ARs)
    return mAP, mAR, F1_scores

mAP, mAR, F1_score = evaluate_model(dataset_val, model, inference_config)
print("Precisión Media mAP (mean average precision): %.4f" % mAP)
print("Recall Promedio mAR: (mean average recall): %.4f" % mAR)

F1_score_2 = (2 * mAP * mAR)/(mAP + mAR)
print('F1-score : ', F1_score_2)

Precisión Media mAP (mean average precision): 0.8420 Recall Promedio mAR: (mean average recall): 0.9192 F1-score : 0.8789187223313649

[geomaticsbetul]

it worked ! @alcarazolabs thank you so much.. one last question. My values are very low.

Precisión Media mAP (mean average precision): 0.3465 Recall Promedio mAR: (mean average recall): 0.5414 F1-score : 0.4225485890183695 My dataset consists of satellite images. You can see my training results below. Do you have any suggestions so I can improve the accuracy?

I made 400 epoks number of test img : 697 number of val img : 97

[alcarazolabs]

@geomaticsbetul provide more images 'dataset_val' I just trained my model with 80 images and 20 images for testing.

[christianburbon]

Hi @alcarazolabs, you have to set USE_MINI_MASK to False in your configs

this worked for me. the gt_mask just had a different shape from the model predicted mask. Thanks @LSnyd

[alcarazolabs]

Hi @alcarazolabs, you have to set USE_MINI_MASK to False in your configs

this worked for me. the gt_mask just had a different shape from the model predicted mask. Thanks @LSnyd

did you had problems with 0 values for precision or recall? I'm getting 0 values in some cases in a new project.

[christianburbon]

Hi @alcarazolabs, you have to set USE_MINI_MASK to False in your configs

this worked for me. the gt_mask just had a different shape from the model predicted mask. Thanks @LSnyd

did you had problems with 0 values for precision or recall? I'm getting 0 values in some cases in a new project.

I have not faced 0 value issues. However, it might be possible that you have some predicted masks that does not have a corresponding ground truth mask.

[kimile599]

it worked ! @alcarazolabs thank you so much.. one last question. My values are very low.

Precisión Media mAP (mean average precision): 0.3465 Recall Promedio mAR: (mean average recall): 0.5414 F1-score : 0.4225485890183695 My dataset consists of satellite images. You can see my training results below. Do you have any suggestions so I can improve the accuracy?

I made 400 epoks number of test img : 697 number of val img : 97

Hi, can you calculate AP, AR, F1 for each category, i am trying to get those numbers but failed.

[DickoAI]

hi @alcarazolabs thanks for your details above (https://github.com/matterport/Mask_RCNN/issues/2474#issuecomment-812651613) I am new to maskrcnn, and although I am managing to train the model (it seems), the model does not tell me the mAP. I only have one class.

My validation json file looks like this: "images": [ { "file_name": "image_9.jpg", "height": 640, "width": 640, "id": 0 }, "area": 8450, "iscrowd": 0, "image_id": 722, "bbox": [ 143, 101, 25, 338 ], "category_id": 0, "id": 1054 }, { "segmentation": [ 64, 284, 73, 284, 73, 348, 64, 348 ], "categories": [ { "supercategory": "none", "id": 0, "name": "Item1" } I keep getting errors when trying to calculate the mAP, I tried the code below, but wondering if there are problems with my tensorflow and keras versions (for inference i am unsing a new environment with versions for tensorflow 1.3.0. and keras 2.0.8.

`import os import sys import numpy as np

Add the path to the mrcnn folder to the system path

sys.path.append('/users/New_MCRNN/Mask_RCNN/mrcnn')

from inferenceconfig import InferenceConfig from mrcnn import model as modellib from mrcnn import utils import json from pycocotools.coco import COCO

ROOT_DIR = os.path.abspath("/users/New_MCRNN/Mask_RCNN/") sys.path.append(ROOT_DIR)

Path to the pre-trained model

MODEL_PATH = '/users/New_MCRNN/Mask_RCNN/logs/log20230913T1112/mask_rcnn_1_0002.h5'

Path to COCO dataset

DATASET_DIR = '/users/Data/Mask_RCNN/val2017' ANNOTATIONS_PATH = '/users/Data/Mask_RCNN/val2017/annotations/instances_val2017.json'

class CocoDataset(utils.Dataset): def load_data(self, annotations_path, dataset_dir): coco = COCO(annotations_path) class_ids = sorted(coco.getCatIds()) image_ids = list(coco.imgs.keys())

    for class_id in class_ids:
        self.add_class("coco", class_id, coco.loadCats(class_id)[0]["name"])
    
    for image_id in image_ids:
        self.add_image(
            "coco",
            image_id=image_id,
            path=os.path.join(dataset_dir, coco.imgs[image_id]['file_name']),
            annotations=coco.loadAnns(coco.getAnnIds(
                imgIds=[image_id],
                catIds=class_ids,
                iscrowd=None)),
            width=coco.imgs[image_id]["width"],
            height=coco.imgs[image_id]["height"]
        )

def load_mask(self, image_id):
    info = self.image_info[image_id]
    annotations = info['annotations']
    count = len(annotations)
    masks = np.zeros([info['height'], info['width'], count], dtype=np.uint8)
    class_ids = np.zeros(count, dtype=np.int32)
    
    for i, annotation in enumerate(annotations):
        class_id = annotation['category_id']
        class_ids[i] = class_id
        masks[:, :, i] = annotation['mask']
    
    return masks, class_ids.astype(np.int32)

Configuration for inference

config = InferenceConfig() config.display()

Create model object in inference mode.

model = modellib.MaskRCNN(mode="inference", model_dir=ROOT_DIR, config=config)

Load trained weights

model.load_weights(MODEL_PATH, by_name=True)

Load validation dataset

dataset = CocoDataset() dataset.load_data(ANNOTATIONS_PATH, DATASET_DIR) dataset.prepare()

Evaluate model on the validation dataset

APs = [] for image_id in dataset.image_ids: image, image_meta, gt_class_id, gt_bbox, gt_mask = modellib.load_image_gt(dataset, config, image_id, use_mini_mask=False) molded_images = np.expand_dims(modellib.mold_image(image, config), 0) results = model.detect([image], verbose=0) r = results[0] AP, _, _, _ = utils.compute_ap(gt_bbox, gt_class_id, gt_mask, r['rois'], r['class_ids'], r['scores'], r['masks']) APs.append(AP)

print("mAP: ", np.mean(APs)) ` any help would be much appreciated. thank you.

[DickoAI]

Actually, I'll start by implementing the function you described above. How you you load the dataset and config to the evaluation function ?