Could not reproduce the DDPM results on LSUN-Church.

[lmxyy]

Hi, I tried using your converted model on LSUN-Church dataset (ema_diffusion_lsun_church_model/model-4432000.ckpt), but I found my FID results did not match the paper results of the original DDPM paper (10.44 vs 7.89). Specifically, I generated 50k samples with DDIM repo. The evaluation protocol is the original DDPM with 1k steps, whose running command is

python main.py --config church.yml --exp ./exp/church-ddpm --doc church --sample --fid --timesteps 1000 --eta 1 --ni --use_pretrained --sample_type ddpm_noisy

I use clean-fid for my FID computation, whose code is

from cleanfid import fid

score = fid.compute_fid('exp/church-ddpm/image_samples/images/',
                        dataset_res=256, dataset_name='lsun_church',
                        dataset_split='train', mode='legacy_tensorflow')
print(score)

This FID computation protocol should be the same as the protocol reported in DDPM paper. But I found my reproduced FID is 10.44, while the reported FID is 7.89. I think this is a large gap.

I wonder if you could reproduce this number. Is there anything wrong with using your model? Thanks!

[Kettukaa]

My apologies, I am not sure how I made my commit connect to this issue. It is unrelated entirely.

[LuChengTHU]

DDIM is different from DDPM, and their generation performance are not the same, even for very large steps.