peylnog
PSNR discrepancy on Set5 x4 benchmark
Hello, and thanks for your innovative work! I truly appreciate you making the code and pretrained models publicly available, this greatly helps the community. While attempting to reproduce the results, I encountered a metric discrepancy I hope you can help clarify:
I'm trying to reproduce the paper results using the provided code and model (ContinuousSR.pth), but I'm getting substantially lower PSNR scores on the Set5 x4 benchmark
My result: 29.8210 PSNR Paper-reported result: 32.93 PSNR
Is the metric calculation (PSNR) implemented differently from common SR benchmarks? I'm not familiar with SR benchmark specifics - could you please clarify or help me solve the potential issues?
Thanks for your time in advance!
Thanks for you interest in our work.
Our metric calculation (PSNR) is implemented in the same manner as common SR benchmarks. Specifically, on the Set5 X4 dataset, we first downscale the HR by a factor of 4 using the bicubic method. We then upscale the downscaled images to the original HR size using ContinuousSR and calculate the PSNR in the Y domain between the super-resolved images and the HR images.
I think the possible reason might be the different ways of generating the LR images. Did you use bicubic to generate the LR images?
Additionally, the Set5 dataset can be found in https://cv.snu.ac.kr/research/EDSR/benchmark.tar.
Thanks for your reply and information!
I tried to reproduce the results by running the command as suggested in the README: python test.py --config ./configs/test/test-set5-4.yaml --model ContinuousSR.pth
As specified by the ./configs/test/test-set5-4.yaml, the dataset is constructed by the wrapper "sr-implicit-downsampled". Thus, the HR images under path "./Set5/HR" will be downscaled by a factor of 4 using the "InterpolationMode.BICUBIC" mode.
After checking the evaluation metric code, I found that the absent parameter "eval_type: benchmark-4" in "test-set5-4.yaml" caused the PSNR discrepancy (the previous value is calculated in the RGB domain instead of the Y domain).
Currently, I have obtained the PSNR result "31.7930" on the Set5 x4 dataset, but there is still about 1 db PSNR gap between the reported result. Could you give me more information? Many thanks!
Thank you very much for your reminder. We indeed missed the definition of the eval_type parameter in "test-set5-4.yaml" when we open-sourced it.
We retested the results on Set5 X4 and obtained a PSNR of 33.17. Although there is a 0.07 dB difference compared to the performance shown in Figure 1(d) of our paper, we believe this difference is reasonable since the weight file is the result of our reproduction.
Did you modify any other parameters in the "test-set5-4.yaml" file during the testing? If not, we suggest you save the super-resolved images to check if there is anything abnormal.
Thank you for your prompt response.
I have not modified any other parameters in the "test-set5-4.yaml" file during the testing.
I have also checked the generated results and, in my opinion, there are no obvious abnormalities. I have attached the results to this issue to help identify any potential issues.
Additionally, I tried running the code with different PyTorch versions (1.13.0 and 2.1.0), but I reproduced the same result in both cases.
Could you kindly provide more suggestions or any additional information?
The image you output does have differences from ours. This is the result after we applied super-resolution using ContinuousSR.
I’m wondering why the output images are different from the ones in your result. Does this mean the provided model weight from Google Drive is not the final trained weight?
Since the source code appears to be almost the same, the input data should also be the same as the files you provided previously (the Set5 HR directory). Given this, I am curious if there are any special environment requirements or additional steps needed to reproduce the exact experiment results.
@KongYuJL I obtained exactly the same results as you during the test, which are somewhat different from those of the paper's author. I would like to ask if you and the author know how to solve this problem now
