About the utilization of DiffBIR for the blind image denoising

[momo1986]

I want to use the APIs of DiffBIR to do the task of blind image denoising as a plug-in in other program.

Is there a demo to call the APIs rather than run the file of inference.py?

Thanks & Regards! Momo

[momo1986]

Here is my code.

swinir: SwinIR = instantiate_from_config(
    OmegaConf.load("ckp/diffbir/configs/inference/swinir.yaml")
)
swinir.load_state_dict(load_model_from_url(MODELS["swinir_realesrgan"]))
swinir.eval().to(device)


# 2. load stage-2 model
cldm: ControlLDM = instantiate_from_config(
    OmegaConf.load("ckp/diffbir/configs/inference/cldm.yaml")
)
# 2.1 load pre-trained SD
sd_weight = load_model_from_url(MODELS["sd_v2.1_zsnr"])
unused, missing = cldm.load_pretrained_sd(sd_weight)
print(
        f"load pretrained stable diffusion, "
        f"unused weights: {unused}, missing weights: {missing}"
        )
# 2.2 load ControlNet
control_weight = load_model_from_url(MODELS["v2.1"])
cldm.load_controlnet_from_ckpt(control_weight)
print("load controlnet weight")
cldm.eval().to(device)      cast_type = {
        "fp32": torch.float32,
        "fp16": torch.float16,
        "bf16": torch.bfloat16,
        }["fp16"]
cldm.cast_dtype(cast_type)

# 3. load noise schedule
diffusion: Diffusion = instantiate_from_config(
    OmegaConf.load("ckp/diffbir/configs/inference/diffusion_v2.1.yaml")
)
diffusion.to(device)
diffusion_pipeline = SwinIRPipeline(swinir, cldm, diffusion, None, device)   img_torch = torch.from_numpy(np.true_divide(img.astype(np.float32), 255.0)).unsqueeze(0)  output = diffusion_pipeline.run(img_torch, 80, 0.55, True, 512, 256, True, 256, True, 256, True, 256, 128, "", "low quality, blurry, low-resolution, noisy, unsharp, weird textures", 4.0, "noise", "ddim", False, 1.0, 0.0, 0.0, 80.0, 0.0, 1, 1)[0]

Here is input denoised image (512*512)

However, the running result of the pipeline is very adjective.

Dear colleagues, could you help me ?

Thanks & regards! Momo