pytorch_diffusion icon indicating copy to clipboard operation
pytorch_diffusion copied to clipboard

Published 20 hours ago • verified publisher icon pesser

Could not reproduce the results on CIFAR-10

Open zwzhang123 opened this issue 3 years ago • 8 comments
trafficstars

I used your provided model-790000.ckpt on DDIM codebase to generate cifar10 images. But I got the FID is 4.818.

Is there any difference?

zwzhang123 avatar May 13 '22 08:05 zwzhang123

how many samples did you generate? FID depends on the number of samples.

jS5t3r avatar May 21 '23 11:05 jS5t3r

how many samples did you generate? FID depends on the number of samples.

Hi, 50000 samples are generated for evaluation.

zwzhang123 avatar May 21 '23 11:05 zwzhang123

well, that ought to be fine.... I know on that on the paper there was a FID something around 3... DDIM is not as good as ddpm

jS5t3r avatar May 21 '23 11:05 jS5t3r

Did you use the "ema_cifar10"?

jS5t3r avatar May 21 '23 11:05 jS5t3r

Did you use the "ema_cifar10"?

It's been a long time and I forgot which model I used. But, if the diffusion model is not trained with ema, its FID should be 12.

After that, I trained a DDPM model from scratch, and the fid is close to that in the paper.

zwzhang123 avatar May 21 '23 11:05 zwzhang123

Would you recommend another repository (maybe with pretrained weights on cifar10 ema)?

jS5t3r avatar May 21 '23 14:05 jS5t3r

Would you recommend another repository (maybe with pretrained weights on cifar10 ema)?

I forget which codebase provides reliable pre-trained DDPM models. You can try PNDM first. If you cannot reproduce the results, you can train a DDPM model from scratch.

https://github.com/luping-liu/PNDM https://github.com/zzw-zwzhang/pytorch-ddpim

zwzhang123 avatar May 21 '23 14:05 zwzhang123

Thanks. Here comes an update. Seen on huggingface.

https://huggingface.co/google/ddpm-cifar10-32/discussions/10#646a1aec96cfe72aef787baf

There is also claimed to use ddim with 100 steps and fid of 4.5.

jS5t3r avatar May 21 '23 14:05 jS5t3r