Having trouble reproduce the score in botswana4 dataset

[hunchteller]

Hello, thank you for the code and work here. When I tried to reproduce the score on the botswana4 dataset, I only got roughly 26db PSNR with the two stage training strategy, and pretraining phase only gets a 16db PSNR score. Do you have any ideas about what went wrong? Thank you.

Furthermore, when I try to test with the pretrained weights and config file you provided, I get a weird score like the below,

When using the final_prediction.mat you provided, the psnr with the gt I generated is 29.84db. My dataset is generated using the Matlab code "process_botswana.m". Do you have any ideas where things went wrong? Thank you.

[hunchteller]

from scipy.io import loadmat
from utils.metrics import PSNR
import numpy as np
import torch
pr = loadmat('Experiments/bots_download/final_prediction.mat')
print(pr.keys())
total_psnr = []
for k in pr.keys():
    if 'botswana' in k:
        pr1 = pr[k]
        pr1 = torch.from_numpy(pr1).permute(2, 0, 1).contiguous()
        k_sub = k.split('_pred')[0]
        gt = loadmat(f'../datasets_win11/botswana4/botswana4/{k_sub}/{k_sub}.mat')
        gt = torch.from_numpy(gt['ref'].astype(np.int32)).permute(2, 0, 1).contiguous()
        psnr = PSNR(pr1, gt)
        total_psnr.append(psnr)
print(total_psnr)

Here is the code I used to evaluate the psnr of the final_prediction.mat you provided and the gt I generated, and the score are like below. [tensor(31.9064), tensor(30.0319), tensor(31.7481), tensor(28.3671), tensor(27.1711), tensor(29.8685)]

[wgcban]

@zha0x Hi, thank you for your question. your evaluation seems correct. once possible reason could be why you are not getting the same result is that, during pre-processing, I have clipped values above 8000 to 8000 for the Botswana dataset. (see https://github.com/wgcban/HyperTransformer/blob/c5bb015ad73e606e31ac1d273bc00fee9c1305b1/configs/config_HSIT.json#L62).

The reason why I clipped the HSI values between [0,8000] is that I observed very less number of pixels have values above 8000 and if we take the maximum value (which is way higher than 8000) to normalize the data, most of the pixel values become very low; resulting it longer training time.

[hunchteller]

@wgcban Thank you for your reply. In the botswana4 dataset, the value seems to be 7365 (8000 is for botswana dataset), and the original mat file is clipped under 7365 in the process_botswana4.m like below: https://github.com/wgcban/HyperTransformer/blob/c5bb015ad73e606e31ac1d273bc00fee9c1305b1/datasets/botswana4/process_botswana.m#L14 I also check my generated patch, the max value is 7365 indeed. so this may not be the clipping problem.

And now my biggest problem is that maybe the dataset I generated is somewhat incorrect (maybe due to different Matlab version, I use 2020b). The PSNR between the provided final_prediction.mat in the dropbox and the gt patch I generated is way below the reported value. It would be very appreciated if you could provide the whole generated dataset, so I can check whether my dataset is wrongly generated. Thank you!

[JingYu22]

Hello, @wgcban ! I have the same question about metrics of botswana4 dataset, as@zha0x has mentioned. After the two-stage-training as you instructed, and the metrics I got is: {"loss": 0.07479449920356274, "cc": 0.9171549081802368, "sam": 3.2851600646972656, "rmse": 0.02308078669011593, "ergas": 0.6585464477539062, "psnr": 27.814123153686523}

I also used the trained model you provided, and I got: {"loss": 0.04965453268960118, "cc": 0.956545352935791, "sam": 2.070371150970459, "rmse": 0.015240391716361046, "ergas": 0.4459594190120697, "psnr": 30.830589294433594}

Both results are far from expectation. The version of my matlab is 2021b. Is it possible that different matlab version affects the result?

[Anywayany]

I have the same question about metrics of botswana4 dataset, . After the two-stage-training as instructed, and the metrics I got is: {"loss": 0.0713572446256876, "cc": 0.9239928126335144, "sam": 2.906182050704956, "rmse": 0.021915223449468613, "ergas": 0.5995544195175171, "psnr": 28.204063415527344}

[rainfalj]

I alao meet the similar questions about metrics of chikusei dataset and pavia dataset under matlab2021a . The PSNR I got from botswana4 dataset is 29.84885597229004db.