murel.bootstrap.pytorch icon indicating copy to clipboard operation
murel.bootstrap.pytorch copied to clipboard

Published 20 hours ago • verified publisher icon Cadene

a question about training with/out VGenome

Open archanray opened this issue 5 years ago • 4 comments

Hi, I trained using VGenome, but the performance on VQA overall went down. I used the supplied yaml as config, does anything needs to be changed?

archanray avatar Jun 26 '19 21:06 archanray

@LousyLory Thanks for your interest :)

  • Which accuracy did you obtain with and without VGenome?
  • On which benchmark (train/val, trainval/test-dev)?
  • Is it the accuracy or the open ended accuracy?

Cadene avatar Jun 26 '19 22:06 Cadene

Hi @Cadene, thanks for the response!

  1. The new model under-performs in each of the categories as well as overall (yes/no, number, others and overall).
  2. test-dev
  3. MSCOCO-Open ended.

The following are the values I got: without-VG with-VG yes/no 83.75 82.95 number 49.3 49.21 others 56.93 57.0 overall 67.11 66.8

I understand that these are within error rates, but the second model under-performs in each and every segment of the VQA challenge on test-dev data.

archanray avatar Jun 27 '19 14:06 archanray

@LousyLory Thanks for this info.

I am sorry for the issue. We developed on pytorch v0.5 and released a clean code on pytorch v1.x. I will make sure that our results can be reproduce easily on this version as well. I will keep you updated.

While I focus in this issue, you could use the pretrained models of https://github.com/Cadene/block.bootstrap.pytorch as a baseline if you need to.

Cadene avatar Jun 28 '19 17:06 Cadene

Cool thanks!!

archanray avatar Jun 28 '19 21:06 archanray