SFM
SFM copied to clipboard
majedelhelou
Artifacts for rgb images
as shown below, left image is a rgb img input, right image is the result of channel-wise frequency mask. There are many artifacts, but this is plausible because different channel have different frequency, so how to remove those artifacts or can we just train with this kind of results.
I agree that this could be due to a chromatic aberration across spectral channels, we studied this in a previous project. But I can see the image is getting more blurry so you are effectively applying a low pass filter with the SFM mask, normally the network will be able to ignore the artifacts we got here.
Well, But those colorful noise is due to data out of bounds, this will lead to output's DCT totally different from expected. No frequency will be masked.
top left is input img, lower left is the masked dct of input, top right is the image recovered from masked dct, there are many data out of bounds, lower right is the dct of result img.
This can result in totally different dct map, can those rgb images be used to train?
It is important to keep in mind that generally a binary frequency masking is matched with a spatial convolution, but all kernels are not necessarily physically realisable, so not over [0,255].
However, do not confuse high frequency components and high frequency content. We tested different configurations with masking noise or not, and changing the type of masks etc. What matters most is erasing the image frequency content. Even if there remains noise frequency content/artifacts. And in your example you can clearly see visually that the image high frequency content was reduced (the image becomes blurry). Having said that, it can be an interesting research direction to explore non-binary SFM masks, having a smoothly decreasing mask instead of a binary one can enable you to avoid these artifacts over your [0,255] range. You can even explore applying SFM in the spatial domain to only have realistic kernels. We went for the binary option for theoretical reasons, and also to have one simple solution across methods.