Mask_RCNN
Mask_RCNN copied to clipboard
matterport
How to calculate a confusion matrix?
Hi guys,
I would like know how to calculate confusion matrix on prediction mask rcnn?
Someone can help me?
@Nicollas21 Is your dataset has single class label or multiple classes? It's little tricky if your dataset has multiple classes because now you have to keep the count of TP, FP, FN for each class when computing the matches per image.
- compute_matches -This function finds out the prediction annotation matches wrt to the respective ground truth on per image basis. This filter-out the prediction based on IoU threshold, but uses masks and not the bbox for comparing the overlaps matches. Here, you need to introduce logic to keep the count on a per class/per image basis of (TP, FP, FN) and then aggregate across all classes for all the images in the dataset
@Nicollas21 were you able to resolve this? I'm trying to do the same and not making much headway :/
i have same question. I want to calculate confusion matrix of predicted output. Can someone help?
@manvirvirk l had worked out some working details on how to calculate confusion matrix in this case, see if this can give you some idea. It's not so legible, still you may pick up some thoughts on which you can build upon conf-matrix-pg-1.jpeg conf-matrix-pg-2.jpeg
@mangalbhaskar did you ever write any code to demo your logic? I was not able to understand much from the notes, but maybe you could give a condensed version of it here.
@gautamchitnis I can give you the conceptual understanding of my notes here.
As the confusion Matrix tells us about the distribution of our predicted values across all the actual outcomes.
The trick in the matterport mask_rcnn case is to construct the 3D array data-structure before you start the prediction over the val or test dataset.
This 3D array is a 2D array (zero array) stacked over the total number of images i.e. (N,H,W). The rows are for ground truth class and the columns as the predicted class, or vice-versa. Each 2D array has dimensions of (H,W), where H is equal to W is equal to Total number of categories plus one. This extra col/row keeps the count of number of FP and FN. In the 3D array, N is total number of images for which you need to run the prediction, ideally equal to the total number of images in your val or test dataset, for which you are calculating the confusion matrix.
Now run the predictions for each Image images. For each image, calculate the FP, FN and match for each objects in that image and record the results by increment the appropriate index of the zero array. This in the end aggregate over total number of predicted images.
All you need to do is:
- construct 3D array (N,H,W)
- create the prediction loop for val/test dataset
- for each image call a function that computes the matches (TP), FP and FN and record count in the 3D stack array
- You will get the confusion matrix per image, which now can be aggregated to get the single matrix or extended to derive the mAP or weighted mAP.
You can get the conceptual working and some code in the following links:
- https://stackoverflow.com/a/30748053 - This is the simplest and best illustration of confusion matrix
- http://www.datasciencesmachinelearning.com/2018/11/confusion-matrix-accuracy-precision.html
- https://supervise.ly/explore/plugins/confusion-matrix-75279/overview
- https://medium.com/moonvision/smart-object-detection-evaluation-with-confusion-matrices-6f2a7c09d4d7
- https://medium.com/@jonathan_hui/map-mean-average-precision-for-object-detection-45c121a31173
@mangalbhaskar thank you for such an in-depth explanation. I think I should be able to come up with something usable with all this info. Will update this issue if I have a breakthrough.
@gautamchitnis @mangalbhaskar @JaviF98 Hello. I hope you are doing okey. Did you manage to solve this problem ? I kinda need this code as soon as possible to evaluate my model on the whole test dataset. Thank you.
Hi, first thing first, you need to add these two functions to utils.py (they are not optimized yet, but they do the work)
Note that the get_jou defined in the source code computes the iuo between a boxe and a list of boxes.
def get_iou(a, b, epsilon=1e-5): """ Given two boxes aandb` defined as a list of four numbers:
[x1,y1,x2,y2]
where:
x1,y1 represent the upper left corner
x2,y2 represent the lower right corner
It returns the Intersect of Union score for these two boxes.
Args:
a: (list of 4 numbers) [x1,y1,x2,y2]
b: (list of 4 numbers) [x1,y1,x2,y2]
epsilon: (float) Small value to prevent division by zero
Returns:
(float) The Intersect of Union score.
"""
# COORDINATES OF THE INTERSECTION BOX
x1 = max(a[0], b[0])
y1 = max(a[1], b[1])
x2 = min(a[2], b[2])
y2 = min(a[3], b[3])
# AREA OF OVERLAP - Area where the boxes intersect
width = (x2 - x1)
height = (y2 - y1)
# handle case where there is NO overlap
if (width<0) or (height <0):
return 0.0
area_overlap = width * height
# COMBINED AREA
area_a = (a[2] - a[0]) * (a[3] - a[1])
area_b = (b[2] - b[0]) * (b[3] - b[1])
area_combined = area_a + area_b - area_overlap
# RATIO OF AREA OF OVERLAP OVER COMBINED AREA
iou = area_overlap / (area_combined+epsilon)
return iou
def gt_pred_lists(gt_class_ids, gt_bboxes, pred_class_ids, pred_bboxes, iou_tresh = 0.5):
"""
Given a list of gt and predicted classes and their boxes,
this function associates the predicted classes to their gt classes using a given Iou (Iou>= 0.5 for example) and returns
two normalized lists of len = N containing the gt and predicted classes,
filling the non-predicted and miss-predicted classes by the background instance (index 0).
Args :
gt_class_ids : list of gt classes of size N1
pred_class_ids : list of predicted classes of size N2
gt_bboxes : list of gt boxes [N1, (x1, y1, x2, y2)]
pred_bboxes : list of pred boxes [N2, (x1, y1, x2, y2)]
Returns :
gt : list of size N
pred : list of size N
"""
#dict containing the state of each gt and predicted class (0 : not associated to any other class, 1 : associated to a classe)
gt_class_ids_ = {'state' : [0*i for i in range(len(gt_class_ids))], "gt_class_ids":list(gt_class_ids)}
pred_class_ids_ = {'state' : [0*i for i in range(len(pred_class_ids))], "pred_class_ids":list(pred_class_ids)}
#the two lists to be returned
pred=[]
gt=[]
for i, gt_class in enumerate(gt_class_ids_["gt_class_ids"]):
for j, pred_class in enumerate(pred_class_ids_['pred_class_ids']):
#check if the gt object is overlapping with a predicted object
if get_iou(gt_bboxes[i], pred_bboxes[j])>=iou_tresh:
#change the state of the gt and predicted class when an overlapping is found
gt_class_ids_['state'][i] = 1
pred_class_ids_['state'][j] = 1
#chack if the overlapping objects are from the same class
if (gt_class == pred_class):
gt.append(gt_class)
pred.append(pred_class)
#if the overlapping objects are not from the same class
else :
gt.append(gt_class)
pred.append(pred_class)
#look for objects that are not predicted (gt objects that dont exists in pred objects)
for i, gt_class in enumerate(gt_class_ids_["gt_class_ids"]):
if gt_class_ids_['state'][i] == 0:
gt.append(gt_class)
pred.append(0)
#match_id += 1
#look for objects that are mispredicted (pred objects that dont exists in gt objects)
for j, pred_class in enumerate(pred_class_ids_["pred_class_ids"]):
if pred_class_ids_['state'][j] == 0:
gt.append(0)
pred.append(pred_class)
return gt, pred
`
Then, you will need to add this lines of code to your notebook after loading the test dataset and the weights for your model :
`import pandas as pd gt_tot = np.array([]) pred_tot = np.array([]) mAP_ = []
for image_id in dataset.image_ids:
#image_id = random.choice(dataset.image_ids)
image, image_meta, gt_class_id, gt_bbox, gt_mask =
modellib.load_image_gt(dataset, config, image_id, use_mini_mask=False)
info = dataset.image_info[image_id]
#print("image ID: {}.{} ({}) {}".format(info["source"], info["id"], image_id,
#dataset.image_reference(image_id)))
# Run object detection
results = model.detect([image], verbose=1)
# Display results
#ax = get_ax(1)
r = results[0]
#visualize.display_instances(image, r['rois'], r['masks'], r['class_ids'],
#dataset.class_names, r['scores'], ax=ax,
#title="Predictions")
#log("gt_class_id", gt_class_id)
#log("gt_bbox", gt_bbox)
#log("gt_mask", gt_mask)
gt, pred = utils.gt_pred_lists(gt_class_id, gt_bbox, r['class_ids'], r['rois'], iou_tresh = 0.4)
gt_tot = np.append(gt_tot, gt)
pred_tot = np.append(pred_tot, pred)
#precision_, recall_, AP_ = utils.compute_precision_recall_map(gt_tot, pred_tot)
AP_, precision_, recall_, overlap_ = utils.compute_ap(gt_bbox, gt_class_id, gt_mask,
r['rois'], r['class_ids'], r['scores'], r['masks'], iou_threshold = 0.4)
print("the actual len of the gt vect is : ", len(gt_tot))
print("the actual len of the pred vect is : ", len(pred_tot))
print("the actual precision is : ", precision_)
print("the actual recall is : ", recall_)
#print("the actual overlaps is : ", overlap_)
mAP_.append(AP_)
print("the actual average precision : ", AP_)
print("the actual mean average precision : ", sum(mAP_)/len(mAP_))
directory = 'output' gt_pred_tot_json = {"gt_tot" : gt_tot.astype(int), "pred_tot" : pred_tot.astype(int)} df = pd.DataFrame(gt_pred_tot_json) import os if not os.path.exists(directory): os.makedirs(directory) df.to_json(os.path.join(directory, 'gt_pred_test.json'))`
This way, you will a json file saved that contains the two vectors of ground truth classes and predicted classes ( including the background class which is not considered in the code of Matterport), sorted by matches.
After obtaining these two vectors, you can easally generate a confusion matrix (use this pretty confusion matrix if you find it attractive
anyone came across how to resolve confusion matrix please let me know?????
@Altimis hey man is there any way you can reformat your code or provide a .py file with your actual solution in it? it looks like you have kinda solved the problem, but the messy formatting and lot of verbose printing is making it a little difficult to understand exactly what parts are supposed to go where. I will still give your code a try and if I get a usable output I will post it here. And the formatting issues are also making it difficult to understand the indentation levels.
@Nandita1996 @gautamchitnis
Hi. the solution I mentioned above works well and you get at the end a confusion matrix for the whole test dataset that has all the important elements (Precision, recall, F1...), you could also compute the Average Precision in the same way that is presented in VOC reference and many papers (Mean Average Precision of each class for the whole test dataset instead of the sum of Average Precision of each image devided by the number of images). @gautamchitnis you are actually right, the formatting is so bad. I will reorganize it.
@Nandita1996 @gautamchitnis
Here is the confusion matrix exmplained. Let me know If you spot any problem.
@waleedka Hello,
I hope you are doing well, could you please add the elements needed to plot a confusion matrix for the whole dataset and compute the mean average precision in the same way used in VOC and many other papers ? All these elements are presented in the notebook above.
Thank you
@Altimis the notebook definitely helped! I was able to generate the confusion matrix using a helper script that you mentioned earlier. I do have some queries regarding what the mAP signifies vs what CF signifies. But maybe that's not a great conversation for this issue thread so will keep it somewhere else. Thanks for your work.
@gautamchitnis You are welcome. @Nicollas21 Could you please close this issue ?
@Altimis ...Hey altimis, first after all, thankyou so much for the confusion matrix. I tried all the step defined by you, i pasted function1 and function2 along with another complete code in my utils.py. But i am getting following errors.
@Altimis ...Hey altimis, first after all, thankyou so much for the confusion matrix. I tried all the step defined by you, i pasted function1 and function2 along with another complete code in my utils.py. But i am getting following errors.
Hi, I'm sorry for the late response. I assume that you managed to find a solution for that from this issue.
Here is the confusion matrix exmplained. Let me know If you spot any problem.
Ohhh. Thank you. Your approach actually works. I can evaluate my model by TP, FP metrics for each classes.
@Altimis ...Hey altimis, first after all, thankyou so much for the confusion matrix. I tried all the step defined by you, i pasted function1 and function2 along with another complete code in my utils.py. But i am getting following errors.
@nandita96 have you solved the problem? i'm facing the same issue....
Hello @Altimis ...this approach would work for a single class model too right? (one class+background)
Hello @Altimis ...this approach would work for a single class model too right? (one class+background)
yes
@Altimis ...Hey altimis, first after all, thankyou so much for the confusion matrix. I tried all the step defined by you, i pasted function1 and function2 along with another complete code in my utils.py. But i am getting following errors.
Hello there, can you tell me how you solve this issue? I am having the same problem as well...The code work perfectly fine with 1 class, but it doesnt work when I have 2 classes... kinda need a solution urgently!
I have a question regarding the gt_pred_lists function: I find it weird, that in both cases here:
#check if the overlapping objects are from the same class
if (gt_class == pred_class):
gt.append(gt_class)
pred.append(pred_class)
#if the overlapping objects are not from the same class
else :
gt.append(gt_class)
pred.append(pred_class)
the same happens in the if and the else. Shouldn't it do something else in the else case? Otherwise it does not make sense to distinguish the cases, right?
