How to reproduce the best result?

[Victarry]

I follow the instructions in README.md, but I can't get the best results as the paper said. Here is what I haved done:

1. Follow the instructions from https://github.com/Maluuba/GeNeVA_datasets/, and generate the data file.
2. Download the pretrained object detector model from https://www.microsoft.com/en-us/research/project/generative-neural-visual-artist-geneva/
3. Use the default parameters and run python geneva/inference/train.py @example_args/iclevr-d-subtract.args

Here is my results in iclevr dataset, I trained this model with 2 NVIDIA TITAN Xp about 4 days:

It seems it's far from the performance in the origin paper.

[tsujuifu]

Hi Victarry,

have you solved the problem? The result of my experiment on i-CLEVR is also about 54% f1-score.

[Victarry]

Hi Victarry,

have you solved the problem? The result of my experiment on i-CLEVR is also about 54% f1-score.

Not yet.
But when I remove the -use_fd flag and keeps -use_fg, I got much better results, about 72% f1-score

[mustafacobann]

Hi friends, I'm also trying to reproduce the scores given in the paper, however current performance of the model is far from those mentioned in the paper. Have you been able to find the reason why the performance is not as good as that of authors?

[yukia18]

I'm also trying to reproduce the scores of i-CLEVR as following the instructions noted in README (I mean I used example_args/iclevr-d-subtract.args), but my result is far from the authors, too.

	AP	AR	F1	RSIM
Paper	92.39	84.72	88.39	74.02
Mine	65.84	40.82	49.45	28.33