MM-Diffusion icon indicating copy to clipboard operation
MM-Diffusion copied to clipboard

How many iterations have the pretrained models been trained for?

Open mayank-git-hub opened this issue 1 year ago • 2 comments

Hello Authors, thank you for the wonderful work!

While trying to reproduce the results for the pretrained models, I found a mismatch between the metrics mentioned in the paper. One of the small bugs that i found in the evaluation script is that instead of multiplying the FAD by 1e4, it has been multiplied by 1e3.

Even ignoring that, the results do not match and I believe it could be because of the difference in the number of iterations that the model has been trained for compared to the one mentioned in the paper.

Can you provide details about the pretrained models that you have shared?

Thank you so much!

mayank-git-hub avatar Apr 17 '24 08:04 mayank-git-hub

Hello Authors, thank you for the wonderful work!

While trying to reproduce the results for the pretrained models, I found a mismatch between the metrics mentioned in the paper. One of the small bugs that i found in the evaluation script is that instead of multiplying the FAD by 1e4, it has been multiplied by 1e3.

Even ignoring that, the results do not match and I believe it could be because of the difference in the number of iterations that the model has been trained for compared to the one mentioned in the paper.

Can you provide details about the pretrained models that you have shared?

Thank you so much!

Hi, I also met this issue. Could I know where you found the 1e4 or 1e3? And how many samples did you use for the FVD evaluation?

kaiw7 avatar May 03 '24 16:05 kaiw7

Could I know where you found the 1e4 or 1e3?

https://github.com/researchmm/MM-Diffusion/blob/1d2d5ad9b47f57e7d300e087af8eb93181da094d/mm_diffusion/evaluator.py#L170C5-L170C43

And how many samples did you use for the FVD evaluation?

2048 (I used the default settings of the repository)

mayank-git-hub avatar May 05 '24 09:05 mayank-git-hub