bounding boxes coordinate fixing in object detection augmentor: pad and crop

[lm-jira]

Some augmentors edit the images without modifying the coordinate of bounding boxes (gt_boxes variables), which is reported by this issue. This pull request changes the coordinate of bounding boxes for those augmentors.

Motivation and Context

Editing images without changing coordinate of bounding boxes results in incorrect data label for training data and affects the accuracy of object detection model

Description

Edit pad and crop data augmentor classes to accept gt_boxes as an input, modify the coordinate of gt_boxes according to the way images are modified, and return the correct information of gt_boxes.

How has this been tested?

I tested it using the same setting as Blueoil's object detection tutorial, which is detecting human face and hand using a part of OpenImagesV4 dataset. I implemented on server25.

Screenshots (if appropriate):

The example of bounding boxes in augmented image is as follows.

Types of changes

• [ ] Bug fix (non-breaking change which fixes an issue)
• [x] New feature / Optimization (non-breaking change which adds functionality)
• [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected)

Checklist:

• [x] My code follows the code style of this project.
• [ ] My change requires a change to the documentation.
• [ ] I have updated the documentation accordingly.

[ruimashita]

@lm-jira Nice!! Please resolve conflicts

[ruimashita]

@lm-jira please implement unit test.

[lm-jira]

@ruimashita I edited the code as follows

• Pad: commit 05898c0157f312cc791808a08a828cb42fa111da The coordinate of bounding box is added by the padding size as suggested by this issue.

• Crop: commit d4a294046c3d21e5595b75622277b1a738754017 The bounding box is removed if it is outside the cropped area. Then, the coordinate is adjusted according to the cropped position. Finally, if a part of the bounding box is outside the cropped image, the coordinate is changes so that it is at the margin of the image. (e.g. if x < 0, update x to be 0)

• Rotate: commit be361963a49ac3bc871a45ced4d7a3cc5c6c0e22 I used this article as a reference for moving the bounding box. The bounding box is changed with the following step.

  1. Find the coordinate of four corner points of the bounding boxes.
  2. Rotate the four points using the image's center as the center of rotating.
  3. Find the smallest bounding box that the rotated points in the previous step can fit in (the black box in the rightmost image) and update the corresponding bounding box to that smallest box.

[iizukak]

@ruimashita @nlpng Please review this PR.

[ruimashita]

@lm-jira How much effect to final result the rotate way ?

[ruimashita]

@lm-jira Could you implement unit test or take screenshots when the changed box is outside of image of pad and crop. Looks not considered or wrong when outside for me.

[ruimashita]

This is not bug, new feature, I think

[lm-jira]

@lm-jira How much effect to final result the rotate way ?

@ruimashita Rotating bounding box results in higher overall accuracy. (Red line = Accuracy of model after rotating bounding box, Pink line = Accuracy of model before rotating the bounding box)

[ruimashita]

@lm-jira Thanks 😀 But this is not final report. This is middle progress.

[lm-jira]

@ruimashita

@lm-jira Could you implement unit test or take screenshots when the changed box is outside of image of pad and crop. Looks not considered or wrong when outside for me.

Do you mean bounding boxes in the images below? (cropping image and cut the bounding box to the margin before padding image)

@lm-jira Thanks 😀 But this is not final report. This is middle progress.

Should I write the report comparing the results in Kibela?? 🤔

[lm-jira]

@ruimashita I write a report comparing the result before and after correcting the rotate augmentor here. This time, I increased the number of epochs from 100 to 600 to be able to clearly see the difference.

[ruimashita]

@lm-jira Thanks. Nice report. But the MAP is low, it is not suitable to actual project, I still wonder the method is useful or not. Could you compare Rotate or not using this config ? https://github.com/blue-oil/blueoil/blob/master/lmnet/configs/core/object_detection/lm_fyolo_quantize_pascalvoc_2007_2012.py

[lm-jira]

@ruimashita I did the comparison experiment of rotate augmenter using pascalvoc_2007 as you mentioned. The difference between before and after modification is very little as shown below. More detail can be found here.

I also simply search the article for rotating bounding box in object detection model but cannot find. I'm thinking that rotating the bounding boxes does not worth the increasing performance so I remove the rotate part from this PR and move it to another branch (may be not create a PR for it).

[a-hanamoto]

@lm-jira How is the situation of this issue? Do you need any help?

[lm-jira]

@a-hanamoto I think the fix of bounding box coordinate for pad and crop augmentation is fine but it doesn't pass the review so I cannot merge it yet. Because the Blueoil has a lot of update, I think it'd be better to make a new PR for this issue.

[a-hanamoto]

@lm-jira Got it. Thanks! Then could you please make a new PR?

[lm-jira]

@a-hanamoto Okay. I will do it when I have time.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
3 out of 10 committers have signed the CLA.

:white_check_mark: lm-lily
:white_check_mark: tkng
:white_check_mark: iizukak
:x: ruimashita
:x: nlpng
:x: hadusam
:x: patrick-su
:x: lm-konda
:x: tfujiwar
:x: lm-jira
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 10 committers have signed the CLA.

:white_check_mark: iizukak
:x: hadusam
:x: tkng
:x: ruimashita
:x: nlpng
:x: lm-lily
:x: patrick-su
:x: lm-konda
:x: lm-jira
:x: tfujiwar
_{You have signed the CLA already but the status is still pending? Let us recheck it.}