Changed downscale interpolation to avoid aliasing

[nathanhubens]

Hi !

I recently used the albumentation library for a Kaggle competition and more particularly the Downscale transform.

After looking at the result the transform gave me I was a little bit surprised:

Result using bilinear interpolation

Result using bicubic interpolation

We can see a lot of artifacts and aliasing happening here.

After checking the source code, I noticed that the same interpolation method was used both for the downscaling part and for the upscaling to the original size part. However, as the OpenCV doc mentions:

cv2.INTER_AREA: resampling using pixel area relation. It may be a preferred method for image decimation, as it gives moire’-free results. But when the image is zoomed, it is similar to the INTER_NEAREST method

This was indeed the case, here are the results of the same image but with cv2.INTER_AREA applied for the downscaling part:

Result using bilinear interpolation

Result using bicubic interpolation

So we can see that now the images are of way better quality and better recreates what actual image resizing might look like, which is the main goal of the data transformation.

[Dipet]

Hi, thanks for PR and interesting investigation.

I am not agree with you here

So we can see that now the images are of way better quality and better recreates what actual image resizing might look like, which is the main goal of the data transformation.

I think in many task we need more artifacts on images, because we need to simulate bad image quality. So I think is better to get possibility to set different interpolation methods for downscaled image. I think we can add new argument, like intepolation_downscale and add info to documentation about this behavior

[nathanhubens]

Hi, thank you for your answer.

I think in many task we need more artifacts on images, because we need to simulate bad image quality.

I agree with that but I don't think the artifacts we can see here are relevant to the intended behavior of the function.

The goal of the Downscale data augmentation is to make the network robust images smaller than what you trained it with, and that will probably be upscaled before inference to meet the expected input size of the network. By using a downscaling interpolation different than INTER_AREA, we introduce artifacts that the network will never see on real data.

I think we can add new argument, like intepolation_downscale and add info to documentation about this behavior

This would partially solve the issue, but IMO the INTER_AREA interpolation should be the default because there are not much use cases where you want your network to be robust to aliasing effects. Or even better, this should be the purpose of another augmentation of albumentation, but not Downscale as it is ambiguous.

[Dipet]

INTER_AREA interpolation should be the default

I agree that this should be the default. But in the current implementation, we can only change upscale behavior, with an argument we can change both of them.

[MichaelMonashev]

I think second interpolation argument adds flexibility. It can be passed via tuple: interpolation=(cv2.INTER_AREA, cv2.INTER_NEAREST)

[nathanhubens]

Ok I can add a second argument ! @MichaelMonashev wouldn't a tuple be a little bit misleading for which interpolation corresponds to which phase (downscale/upscale) ? Also, as you'd want to have cv2.INTER_AREA for downscaling 99% of the time, it probably should be a default that you don't need to pass every time you call the function, what do you think of simply ?

def downscale(img, scale, down_interpolation = cv2.INTER_AREA, up_interpolation=cv2.INTER_NEAREST):

[BloodAxe]

+1 for having two interpolation arguments

чт, 30 апр. 2020 г. в 4:44 PM, Nathan Hubens [email protected]:

Ok I can add a second argument ! @MichaelMonashev https://github.com/MichaelMonashev wouldn't a tuple be a little bit misleading for which interpolation corresponds to which phase (downscale/upscale) ? Also, as you'd want to have cv2.INTER_AREA for downscaling 99% of the type, it probably should be a default that you don't need to pass every time you call the function, what do you think of simply ?

def downscale(img, scale, down_interpolation = cv2.INTER_AREA, up_interpolation=cv2.INTER_NEAREST):

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/albumentations-team/albumentations/pull/584#issuecomment-621861419, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAEB6YHOZEVRHAKJJT5IB2LRPF6CHANCNFSM4MDISQJA .

[nathanhubens]

I've updated the transform code, it obviously fails at some tests since they are outdated, should I change them accordingly too ?

[MichaelMonashev]

@BloodAxe , the new downscale() interface is incompatible with the previous one. Is it ok?

[BloodAxe]

Whenever possible, we do not want to change existing behavior of augmentations. That means, newly introduced down_interpolation and up_interpolation should have default values as they were before this PR. e.g INTER_NEAREST. Obeying this rule causes some friction, but this way prevents from causing silently changes in training results. Imagine one has training pipeline with Downscale augmentation. After updating to never release he may gets totally different model performance since we have changed defaults. This should be avoided at all cost. Therefore, when possible - we should stick to defaults.

At the same time, I agree that default interpolation mode is sub-optimal. So here's my suggestions:

1. Let's make the following change in c'tor:

def __init__(self, scale_min=0.25, scale_max=0.25, down_interpolation=None, up_interpolation=None, always_apply=False, p=0.5):
  if down_interpolation is None:
    down_interpolation = cv2.INTER_NEAREST
    warnings.warn("Using default interpolation INTER_NEAREST, which is sub-optimal. Please specify down_interpolation mode to Downscale explicitly. "

  if up_interpolation is None:
    up_interpolation= cv2.INTER_NEAREST
    warnings.warn("Using default interpolation INTER_NEAREST, which is sub-optimal. Please specify up_interpolation mode to Downscale explicitly. "

In this way we a) Inform user that he can change interpolation to optimal values b) Keep default behavior intact

I've updated the transform code, it obviously fails at some tests since they are outdated, should I change them accordingly too ?

Pulling latest changes to your PR branch from master should resolve the issues

[Dipet]

Also be cool if we will provide url to this PR in warning message for more information.

[Dipet]

I think we should not rename or remove interpolation argument for backward compatibility. At this time our test failed for this reasons and for somebody will fail old pipelines.

I think we need to leave in place interpolation and add warning:

if interpolation is not None and down_interpolation is None and up_interpolation is None:
    down_interpolation = cv2.INTER_NEAREST
    up_interpolation = down_interpolation
    warnings.DeprecationWarning("Argument interpolations is deprecated. Please specify interpolation mode directly for downscale and upscale. ")

[MichaelMonashev]

We can keep back compibility if change down_interpolation and up_interpolation to interpolation=(cv2.INTER_AREA, cv2.INTER_NEAREST). And provide documentation about first and second interpolation.

[Dipet]

Yes, I agree with @MichaelMonashev If we will use interpolation as tuple we are will avoid all this problems. For such case we can use this checks:

if interpolation is None:
    self.down_interpolation, self.up_interpolation = cv2.INTER_NEAREST, cv2.INTER_NEAREST
    warnings.warn("Using default interpolation INTER_NEAREST, which is sub-optimal."
                  "Please specify interpolation mode for downscale and upscale explicitly."
                  "For additional information see this PR https://github.com/albumentations-team/albumentations/pull/584")
elif isinstance(interpolation, int):
    self.down_interpolation, self.up_interpolation = interpolation, interpolation
else:
    self.down_interpolation, self.up_interpolatio = interpolation

[nathanhubens]

Hi ! Sorry for the delay,

I've added the interpolation argument as a tuple, tell me what you think about it.

Cheers,