Duplication artifacts in resulting warped image

[AllenJRZhang]

Hi! Thank you for providing this implementation! Following your instructions however I noticed that the resulting warped image (I slightly tweaked your code so that run.py generates a warped image using your backwarp function) contains some duplication artifacts, as shown below:

Original Example:

Warped Image Result (Look at the blade):

Another Example:

Warped Image Result:

This did not occur with traditional optical flow estimation methods I tried or your pytorch-unflow implementation. (haven't tested the rest but will!) I wonder if it's a problem of PWC-Net itself, or this implementation specifically, or something with pytorch? Also I'd really appreciate it if you could share some insights regarding how to alleviate this problem. Thanks!

[sniklaus]

I am under the impression that that is expected due to occlusion. Consider the first image with the blade moving from left to right, there is an area on the right side of the blade that is not visible in the second frame. Yet, it still has optical flow values that point to where that area should be in the second frame. However, in the second frame that area is occluded by the blade which has moved slightly to the right. This will then lead to the blade being copied twice during backward warping. There are multiple approaches to address these, for example you could look at the forward-backward consistency of flows and only warp pixels where it is consistent (leaving you with holes in the warped result).

[AllenJRZhang]

Thank you for the reply! I'm relatively new to optical flow in general. I wonder if you could point me in the right direction regarding the "multiple approaches" you mentioned? Some keywords that let me know what to look for would be nice. Also, are these approaches mainly during the warping, or perhaps post processing? And how effective are they? My goal is to make, say, image 2 as structurally close to image 1 as possible (then fuse them together), and artifacts such as duplication or holes are unacceptable.

Another issue is that I also tested your other implementations for calculating optical flow with the same images, liteflownet also has similar problem, unflow net however, does not (albeit introducing other problems which are not in the scope of this discussion, and spynet results are generally poor so I leave that one what). Since the framework for the two are more or less identical except for the network structure, which lead me to speculate that this is perhaps due to the model itself. I also tested traditional approaches to calculate optical flow, for example the Pyflow library (Python wrapper of the Coarse2Fine method by Ce Liu), and did not observe similar issues.

Below I attach the visualizations of the estimated flow for my example 2 by your PWC-Net and Unflow Net resepectively, for a more direct comparison.

PWC-Net

Unflow Net

As you can see, the "activated regions" are slightly larger in Unflow Net than PWC (albeit PWC provides more clean results in general). Could this the the cause for such duplication error? If so, does that mean the most effective option would be changing the model structure and/or fine-tuning the model (which I unfornately cannot afford to do so).

Thanks again for the help!

[sniklaus]

My goal is to make, say, image 2 as structurally close to image 1 as possible (then fuse them together), and artifacts such as duplication or holes are unacceptable.

This is a classic "align and merge" problem, see the align and the merge section here for example.

liteflownet also has similar problem

This is not a problem, this is what you want. Otherwise the flow isn't correct. Maybe start with examples where you have the ground truth optical flow so you get a feel for how it is supposed to be.

[sniklaus]

Closing this issue for now, feel free to reopen should you have any more concerns.