Resize mask should not use nearest method of opencv

[shuyangyue]

I'm using this project to train my segmentation model. I find that the mask has a right-down offset to the image. Because the opencv resize_nearest is wrong. Please refer the opencv project issue:
https://github.com/opencv/opencv/issues/9096 https://github.com/opencv/opencv/issues/10146

The code of opencv is: for( x = 0; x < dsize.width; x++ ) { int sx = cvFloor(x*ifx); x_ofs[x] = std::min(sx, ssize.width-1)*pix_size; } It's not the nearest, so it will be a offset between image and mask.

Mask uses the resize nearest in Pillow has no offset to the image. Using the resize nearest of opencv to train segmentation model will have a bad result.

[BloodAxe]

That's a good point which is worth investigation. From what I know, our library were key component in winning many image segmentation competitions and getting SOTA results on some segmentation benchmarks. Therefore even if there is an issue with NN interpolation of masks, the impact on trained model accuracy is not so obvious. Since one pixel offset is rarely the biggest problem.

I can think of having nearest-pillow interpolation mode, but the effort in implementing this feature is not clear at this point. There are lot of usages of OpenCV interpolation under the hood in many augmentations. Would it be possible to you to run ablation study and demonstrate how big the change in accuracy of the segmentation mask for OpenCV and Pillow interpolation modes are?

[shuyangyue]

Thank you for your reply. There is no effect on the iou accuracy of segmentation model. Because mask using opencv resize_nearest causes the ground truth has an offset. If I use the model to predict a image, the result will have about one pixel offset in my model. Examples are below. Left is using the origin albumentations library. Right is pillow resize nearest mode. It's more obvious when resize mask to a very small size. If input is large, it will be not obvious. That is if resize scale is mall, it will not obvious, if resize scale is large, it will be obvious . Resize-nearest in pillow or opencv will have no effect on the iou accuracy, because it affects the ground truth.

[BloodAxe]

It seems we can use cv2.INTER_NEAREST_EXACT interpolation mode to be consistent with PIL, scikit-image or Matlab. @Dipet @creafz @ternaus what are your thoughts on this?

[ternaus]

Works for me.

[creafz]

I do agree that we need a way to fix the current behavior of OpenCV's nearest interpolation that we use for masks.

However, we need a way to inform the current users of different behaviors of cv2.INTER_NEAREST and cv2.INTER_NEAREST_EXACT and let them choose the one they prefer. By following this path, we will not introduce a sudden significant change that will break the current behavior of augmentation pipelines (even if the current behavior is not optimal and may be seen as 'buggy', it is still how Albumentations resizes masks since the first release).

So my proposal is the following:

1. Add a way for users to specify the interpolation algorithm for masks. For example, we could use a parameter for A.Compose, e.g.

transform = A.Compose([...], mask_interpolation=cv2.INTER_NEAREST_EXACT)

2. In the nearest (no pun intended) release of Albumentations, add a warning which says that the next release will use cv2.INTER_NEAREST_EXACT instead of cv2.INTER_NEAREST for mask interpolation, and to preserve the old behavior, the user must explicitly specify the mask interpolation method for an augmentation pipeline. Let's make a separate article in the documentation that tells the difference between those interpolation methods.
3. After that, in a new release, we will change the default interpolation method to cv2.INTER_NEAREST_EXACT, but still, show a warning if the user didn't specify the interpolation method and Albumentations used the default value.

[aksg87]

@creafz do you know if mask_interpolation is going to be added? Any recommendations on current best practice for most segmentation problems?

[creafz]

Hey, @aksg87

We are working on it, but currently, I don't have an ETA for this feature. As a baseline, I think you could stick to the current behavior of Albumentations, which seems to work reasonably well in practice.

[aksg87]

Hey, @aksg87

We are working on it, but currently, I don't have an ETA for this feature. As a baseline, I think you could stick to the current behavior of Albumentations, which seems to work reasonably well in practice.

Great! Yes I have used the baseline with a lot of success. Just curious if I should change default types for some optimization

[karthikpullalarevu]

Has this issue been solved?

[Dipet]

Unfortunately, this issue has not been fixed yet.

[karthikpullalarevu]

Can I try to fix it?

On Tue, Aug 17, 2021, 10:20 PM Mikhail Druzhinin @.***> wrote:

Unfortunately, this issue has not been fixed yet.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/albumentations-team/albumentations/issues/850#issuecomment-900461886, or unsubscribe https://github.com/notifications/unsubscribe-auth/ALPCLOOA4RNVQB7XAKMYJULT5KHNJANCNFSM4YTPK54A . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

[zhangwenwen]

Maybe it can be done by replacing mask_new = cv2.resize(mask_old, (w, h), cv2.INTER_NEAREST) with mask_new = cv2.resize(mask_old, (w, h), interpolation=cv2.INTER_NEAREST), but I have not found the reason yet.

[zhangwenwen]

Code:

mask = np.load(label['mask_file'])
mask1 = cv2.resize(mask, (600,600), interpolation=cv2.INTER_NEAREST)
mask2 = cv2.resize(mask, (600,600), cv2.INTER_NEAREST)

Results: