Framework results

[huynguyenbao]

Have you trained this framework? Do the results match with the reported results in the paper?

[Wenchao-Du]

i didn't validate the results reported in original paper, i have only tried it on my own dataset. but this method is truly effective from my observation, but it is only adapted for the zero-mean noise distribution prior. As reported in the paper that it is a extension of the noise2noise.

[huynguyenbao]

Yeah, it is truly effective, but following their setting, my training result only gets 49.55 dB PSNR in the validation dataset (lower than theirs 51.06 dB). And about theory in the paper, I think it is not true mathematically.

[qihuiqing]

Yeah, it is truly effective, but following their setting, my training result only gets 49.55 dB PSNR in the validation dataset (lower than theirs 51.06 dB). And about theory in the paper, I think it is not true mathematically.

i also read this paper recently and have some doubts on theory in the paper, can i discuss with you about it? My email is [email protected]. looking forward to hear from you!

[huynguyenbao]

Yeah, it is truly effective, but following their setting, my training result only gets 49.55 dB PSNR in the validation dataset (lower than theirs 51.06 dB). And about theory in the paper, I think it is not true mathematically.

i also read this paper recently and have some doubts on theory in the paper, can i discuss with you about it? My email is [email protected]. looking forward to hear from you!

Yeah, sure.

[qihuiqing]

Nice to discuss with you, what is your doubts on this paper? I think the formula (2) in the paper is not quite right? how about you?

Huy Nguyen @.***> 于2021年4月17日周六 下午9:25写道：

Yeah, sure.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Wenchao-Du/Neighbor2Neighbor_Pytorch/issues/1#issuecomment-821822316, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKRPYBPXYD3NFGVWFYJMJ7DTJGD4XANCNFSM42Y34GUA .

[huynguyenbao]

Yeah, I think the main ideal random selection in a block 2x2 is not true mathematically. From a statistical perspective, the noise depends on pixel intensity and maybe independent with other pixels. Assume the denoising function is f and 2 neighbours in block 2x2 are y1 and y2. Their loss function is |f(y1) - y2|_2 but the ground truth or original signals of y1 and y2 are x1 and x2 and x1 is different with x2. So that I think it is not true, regardless of their addition regularization term.

[qihuiqing]

I think carefully about what you said. there are some assumptions which is not shown in the paper, i recommond you to review the article Noise2Noise. The Neighbor2Neighbor is a extension on the Noies2Noise and you will find the answer to your question.

Huy Nguyen @.***> 于2021年4月17日周六 下午10:15写道：

Yeah, I think the main ideal random selection in a block 2x2 is not true mathematically. From a statistical perspective, the noise depends on pixel intensity and maybe independent with other pixels. Assume the denoising function is f and 2 neighbours in block 2x2 are y1 and y2. Their loss function is |f(y1) - y2|_2 but the ground truth or original signals of y1 and y2 are x1 and x2 and x1 is different with x2. So that I think it is not true, regardless of their addition regularization term.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Wenchao-Du/Neighbor2Neighbor_Pytorch/issues/1#issuecomment-821829393, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKRPYBKQ2X2J3I4KXP6OCFTTJGJXXANCNFSM42Y34GUA .

[huynguyenbao]

Yeah, I have read it before, noise2noise theory is only true when 2 noisy pixels in the same location have the same ground truth pixel. Because it needs 2 noisy images same scene, which is very hard in a real situation, so they had proposed many noise2noise variants like noise2sim, noiser2noise for using only 1 noisy image.

[qihuiqing]

i argree with you, if the block 2x2 pixels have big difference from each other, the theory of Neighbor2Neighbor is not true. but in the experiments the four sub-samples have no diference intuitively.

Huy Nguyen @.***> 于2021年4月20日周二 下午5:15写道：

Yeah, I have read it before, noise2noise theory is only true when 2 noisy pixels in the same location have the same ground truth pixel. Because it needs 2 noisy images same scene, which is very hard in a real situation, so they had proposed many noise2noise variants like noise2sim, noiser2noise for using only 1 noisy image.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/Wenchao-Du/Neighbor2Neighbor_Pytorch/issues/1#issuecomment-823119476, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKRPYBOA4IK5LWUKXIQUSW3TJVA4PANCNFSM42Y34GUA .

[huynguyenbao]

Yeah, surprisingly the result of neighbor2neighbor is pretty good although without ground truth. Have you tested this theory with the other complex CNNs?

[Wenchao-Du]

Yeah, surprisingly the result of neighbor2neighbor is pretty good although without ground truth. Have you tested this theory with the other complex CNNs?

yes, i have tested it with REDNET-30, which presented a little better effects based on my datasets. Original network in this paper used the blind-spot network, which is a extension of the UNET.

[leonmakise]

Yeah, it is truly effective, but following their setting, my training result only gets 49.55 dB PSNR in the validation dataset (lower than theirs 51.06 dB). And about theory in the paper, I think it is not true mathematically.

Sir, for SIDD dataset, do you input the full size of images or crop them to a certain size? I cannot reproduce those results. Thanks a lot!

[huynguyenbao]

I use random cropping.

[madfff]

Yeah, it is truly effective, but following their setting, my training result only gets 49.55 dB PSNR in the validation dataset (lower than theirs 51.06 dB). And about theory in the paper, I think it is not true mathematically.

Sorry to bother you, sir. I also cannot reproduce those results and find there is little data-processing code on github. I use my data-processing code and official training code. Could you point out my mistake? ps: BayerUnifyAug comes from https://github.com/Jiaming-Liu/BayerUnifyAug

NOISY_PATH = ['_NOISY_RAW_010.MAT','_NOISY_RAW_011.MAT']

MODEL_BAYER = {'GP':'BGGR','IP':'RGGB','S6':'GRBG','N6':'BGGR','G4':'BGGR'}

TARGET_PATTERN = 'RGGB'

class Dataset(data.Dataset):
    def __init__(self, path, crop_size, is_train=True):
        super(Dataset, self).__init__()
        self.crop_size = crop_size
        self.is_train = is_train
        
        if self.is_train:
            self.unify_mode = 'crop'
        else:
            self.unify_mode = 'pad'
        self.file_lists = []
        folder_names = os.listdir(os.path.join(path,'Data'))
        
        for folder_name in folder_names:
            scene_instance_number,scene_number,_ = meta_read(folder_name)
            for file_path in NOISY_PATH:
                self.file_lists.append(os.path.join(path, 'Data', folder_name, scene_instance_number+file_path))
    
    def __getitem__(self, index):
        noisy_path = self.file_lists[index]
        gt_path = noisy_path.replace('NOISY', 'GT')
        _,_,bayer_pattern = meta_read(noisy_path.split('/')[-2])
        
        noisy = h5py_loadmat(noisy_path)
        noisy = BayerUnifyAug.bayer_unify(noisy, bayer_pattern, TARGET_PATTERN, self.unify_mode)
        
        gt = h5py_loadmat(gt_path)
        gt = BayerUnifyAug.bayer_unify(gt, bayer_pattern, TARGET_PATTERN, self.unify_mode)
        
        if self.is_train:
            augment = np.random.rand(3) > 0.5
            noisy = BayerUnifyAug.bayer_aug(noisy, augment[0], augment[1], augment[2], TARGET_PATTERN)
            gt = BayerUnifyAug.bayer_aug(gt, augment[0], augment[1], augment[2], TARGET_PATTERN)

        noisy = pack_raw_np(noisy[:,:,None])
        gt = pack_raw_np(gt[:,:,None])
        
        if self.crop_size[0] != 0 and self.crop_size[1] != 0:
            H, W, _ = noisy.shape
            rnd_h = random.randint(0, max(0, H - self.crop_size[0]))
            rnd_w = random.randint(0, max(0, W - self.crop_size[1]))
            
            noisy = noisy[rnd_h:rnd_h + self.crop_size[0], rnd_w:rnd_w + self.crop_size[1], :]
            gt = gt[rnd_h:rnd_h + self.crop_size[0], rnd_w:rnd_w + self.crop_size[1], :]
        
        noisy = torch.from_numpy(noisy.transpose(2, 0, 1))
        gt = torch.from_numpy(gt.transpose(2, 0, 1))
        return noisy, gt, bayer_pattern
    
    def __len__(self):
        return len(self.file_lists)

def meta_read(info):
    info = info.split('_')
    scene_instance_number       = info[0]
    scene_number                = info[1]
    smartphone_code             = info[2]
    #ISO_level                   = info[3]
    #shutter_speed               = info[4]
    #illuminant_temperature      = info[5]
    #illuminant_brightness_code  = info[6]

    return scene_instance_number,scene_number,MODEL_BAYER[smartphone_code]

def pack_raw_np(im):
    img_shape = im.shape
    H = img_shape[0]
    W = img_shape[1]
    ## R G G B
    out = np.concatenate((im[0:H:2,0:W:2,:], 
                       im[0:H:2,1:W:2,:],
                       im[1:H:2,0:W:2,:],
                       im[1:H:2,1:W:2,:]), axis=2)
    return out

def h5py_loadmat(file_path:str):
    with h5py.File(file_path, 'r') as f:
        return np.array(f.get('x'),dtype=np.float32)

[huynguyenbao]

I cropped a patch that had size 512x512 with stride 128 (overlap between patches) from the raw dataset. Then I had randomly cropped a small patch like 128x128 from the patch 512x512 above. I also did random flipping X and Y.

[madfff]

Thanks for your detailed reply. I directly crop a big patch (512x512) to train the model and it works well.