VESPCN-tensorflow
VESPCN-tensorflow copied to clipboard
JuheonYi
Hello
Recently I was looking at the VESPCN of paper, but it did not provide code. I cannot understand the formula in section 2.3 of paper. See your code on github. Can you ask about it? Can you explain the formula of Spatial transformer motion compensation please? Thank you! ! My e-mail: [email protected]
Hello, sorry for the terrible delay. I missed the issue you left.
From what I understand, equation (5) in section 2.3 means that the pixel value at (x,y) in transformed frame (I_(t+1)') is obtained by reading the pixel value at (x+delta(x), y+delta(y)), where delta is the optical flow between I_t and I_(t+1)
Plus, it seems that (x+delta(x), y+delta(y)) can be out of the range of original image plane, so there can be some pixels where the values cannot be derived. The outmost bicubic interpolation seems to be necessary to fill such blank spots.
Actually, the my current motion compensation implementation block does not seem to work as I expected, and I plan to debug it. It would be a great help if you could share with me any progress!
Best regards, Juheon
Some things confused me too!
As the paper saied, "Video samples are broken into non-overlapping subsamples of spatial dimensions 33×33", the size of one batch is 33*33 which is odd. But the flow estimation, output of the motion compensation, is end with Sub-pixel upscale by x2(or x4) which is even. When training followed the paper and warping the data using the flow above, it would got wrong that dimension was not matched.
More precisely, when the size of training data is 33 * 33, the output of Coarse flow and Fine flow would be 36 * 36 and 34 * 34, which is not match the input size 33*33.
I e-mail to the author about this question but got no answer!
When I changing the size of input with 36*36, the output the flow estimation would be 36 *36 so that I can warp the picture using this flow. But the after a week training, the result is still not as good as the paper claims, 2dB PSNR less than the paper says. Even the network with no MC, which is a simple extension of the ESPCN.
I have no idea about reproducing VESPCN. Has anyone reproduce VESPCN successfully?
