SPMC_VideoSR
SPMC_VideoSR copied to clipboard
jiangsutx
can't find details about motion compensation transformer .
I can‘t find the details of ’MCT mentioned in SPMC and VESPCN . Does MCT refer to spatial transformer networks（STN）？
It is a network, that takes 2 frames as input, and outputs estimated optical flow. The structure is the same as the optical flow module in the paper 'Real-Time Video Super-Resolution with Spatio-Temporal Networks'.
Thank you so much. How can I find the code about the MCTnet? I'm trying to write the net in pytorch. I'm really confused on the warp step using optical flow. How to calculate the backward of warp step?
I tried to implement it on Pytorch. I find that the coarse flow feature map obtained using sub-pixel upscale is 36X36 while the training patch is 33X33. And the size of the coarse flow feature map is alway even, and the size of patch is odd. So it is impossible to cancat the coarse flow and the images for calculating fine flow. How do you solve this? Would you please give me more details about your motion compensation net? The padding and the size change during convolution.
I implement warp operator based on the code here: https://github.com/tensorflow/models/blob/master/research/transformer/spatial_transformer.py#L145 The difference is I modify parameter theta to flow field.
For the second question, I am a little confused. For our experiments, we use even number for the input patch size, such as 400 512. By doing this, when we do downsample (stride 2) and upsample (deconvolution), the output size will be exactly the same as input.
Thank you very much! I'm trying to evaluate vid4 on SPMC. If is possible, would you please provide test code of scale X3 and frame X3 for vid4 dataset? I only find average PSNR on the paper.
Hi @jiangsutx ,
Could you please share your modified version of the spatial transformer that accetps the optical flow field as a parameter?
Sorry for asking again. If is possible, would you please provide test code of scale X3 and frame X3 for vid4 data set? I’m making a comparison of video SR. it would be thankful for uploading those models.