facebookresearch
features from x-152pp pretrained model doesn't recreate the "movie_mcan.grid.vqa2_vg" test-dev results
I am trying to extract the features for test split of Coco dataset. I used the pretrained model of Backbone X-152++ in https://github.com/facebookresearch/grid-feats-vqa
to extract features I used python3.6 extract_grid_feature.py --config-file configs/X-152-challenge.yaml --dataset coco_2015_test MODEL.WEIGHTS /home/grid-feats-vqa-master/output/X-152pp.pth
and then later on tested with pretrained model of movie_mcan CUDA_VISIBLE_DEVICES=0,1 mmf_predict config=projects/movie_mcan/configs/vqa2/defaults.yaml \model=movie_mcan \dataset=vqa2 \run_type=test checkpoint.resume_zoo=movie_mcan.grid.vqa2_vg \training.num_workers=2 \training.batch_size=32
I got the following results upon submission [{"test-dev": {"yes/no": 83.06, "number": 34.09, "other": 56.85, "overall": 65.06}}]
that is too low then the stated results testdev accuracy - 73.92%
please if you could tell what could be the possible reason for that?
When you ran the mmf_predict command that you mentioned, some logs must have been produced to stdout. Can you provide those logs?
@apsdehal Here are the logs
(mmf) $ CUDA_VISIBLE_DEVICES=0,1 mmf_predict config=projects/movie_mcan/configs/vqa2/defaults.yaml \model=movie_mcan \dataset=vqa2 \run_type=test checkpoint.resume_zoo=movie_mcan.grid.vqa2_vg \training.num_workers=2 \training.batch_size=32 2021-04-17T21:16:00 | mmf.utils.configuration: Overriding option config to projects/movie_mcan/configs/vqa2/defaults.yaml 2021-04-17T21:16:00 | mmf.utils.configuration: Overriding option model to movie_mcan 2021-04-17T21:16:00 | mmf.utils.configuration: Overriding option datasets to vqa2 2021-04-17T21:16:00 | mmf.utils.configuration: Overriding option run_type to test 2021-04-17T21:16:00 | mmf.utils.configuration: Overriding option checkpoint.resume_zoo to movie_mcan.grid.vqa2_vg 2021-04-17T21:16:00 | mmf.utils.configuration: Overriding option training.num_workers to 2 2021-04-17T21:16:00 | mmf.utils.configuration: Overriding option training.batch_size to 32 2021-04-17T21:16:00 | mmf.utils.configuration: Overriding option evaluation.predict to true 2021-04-17T21:16:02 | mmf.utils.distributed: Distributed Init (Rank 1): tcp://localhost:11890 2021-04-17T21:16:02 | mmf.utils.distributed: Distributed Init (Rank 0): tcp://localhost:11890 [W ProcessGroupGloo.cpp:471] Warning: Unable to resolve hostname to a (local) address. Using the loopback address as fallback. Manually set the network interface to bind to with GLOO_SOCKET_IFNAME. (function operator()) [W ProcessGroupGloo.cpp:471] Warning: Unable to resolve hostname to a (local) address. Using the loopback address as fallback. Manually set the network interface to bind to with GLOO_SOCKET_IFNAME. (function operator()) 2021-04-17T21:16:04 | mmf.utils.distributed: Initialized Host 127.0.0.1localhost.localdomainlocalhost as Rank 0 2021-04-17T21:16:04 | mmf.utils.distributed: Initialized Host 127.0.0.1localhost.localdomainlocalhost as Rank 1 2021-04-17T21:16:17 | mmf: Logging to: ./save/train.log 2021-04-17T21:16:17 | mmf_cli.run: Namespace(config_override=None, local_rank=None, opts=['config=projects/movie_mcan/configs/vqa2/defaults.yaml', 'model=movie_mcan', 'dataset=vqa2', 'run_type=test', 'checkpoint.resume_zoo=movie_mcan.grid.vqa2_vg', 'training.num_workers=2', 'training.batch_size=32', 'evaluation.predict=true']) 2021-04-17T21:16:17 | mmf_cli.run: Torch version: 1.6.0+cu101 2021-04-17T21:16:17 | mmf.utils.general: CUDA Device 0 is: Tesla K80 2021-04-17T21:16:17 | mmf_cli.run: Using seed 17054846 2021-04-17T21:16:17 | mmf.trainers.mmf_trainer: Loading datasets 2021-04-17T21:16:25 | torchtext.vocab: Loading vectors from /home/ifrahmaqsood/.cache/torch/mmf/glove.6B.300d.txt.pt 2021-04-17T21:16:30 | torchtext.vocab: Loading vectors from /home/ifrahmaqsood/.cache/torch/mmf/glove.6B.300d.txt.pt 2021-04-17T21:16:44 | mmf.trainers.mmf_trainer: Loading model 2021-04-17T21:16:56 | mmf.trainers.mmf_trainer: Loading optimizer 2021-04-17T21:16:56 | mmf.trainers.mmf_trainer: Loading metrics WARNING 2021-04-17T21:16:56 | py.warnings: /home/ifrahmaqsood/new-mmf/mmf-master/mmf/utils/distributed.py:272: UserWarning: cfg.pretty() is deprecated and will be removed in a future version. Use OmegaConf.to_yaml(cfg)
builtin_warn(*args, **kwargs)
WARNING 2021-04-17T21:16:56 | py.warnings: /home/ifrahmaqsood/new-mmf/mmf-master/mmf/utils/distributed.py:272: UserWarning: cfg.pretty() is deprecated and will be removed in a future version. Use OmegaConf.to_yaml(cfg)
builtin_warn(*args, **kwargs)
2021-04-17T21:16:56 | mmf.utils.checkpoint: Loading checkpoint WARNING 2021-04-17T21:17:20 | mmf: Key data_parallel is not present in registry, returning default value of None WARNING 2021-04-17T21:17:20 | mmf: Key distributed is not present in registry, returning default value of None WARNING 2021-04-17T21:17:20 | mmf: Key data_parallel is not present in registry, returning default value of None WARNING 2021-04-17T21:17:20 | mmf: Key distributed is not present in registry, returning default value of None WARNING 2021-04-17T21:17:34 | py.warnings: /home/ifrahmaqsood/new-mmf/mmf-master/mmf/utils/distributed.py:272: UserWarning: 'optimizer' key is not present in the checkpoint asked to be loaded. Skipping. builtin_warn(*args, **kwargs)
WARNING 2021-04-17T21:17:34 | py.warnings: /home/ifrahmaqsood/new-mmf/mmf-master/mmf/utils/distributed.py:272: UserWarning: 'optimizer' key is not present in the checkpoint asked to be loaded. Skipping. builtin_warn(*args, **kwargs)
WARNING 2021-04-17T21:17:35 | py.warnings: /home/ifrahmaqsood/new-mmf/mmf-master/mmf/utils/distributed.py:272: UserWarning: 'lr_scheduler' key is not present in the checkpoint asked to be loaded. Setting lr_scheduler's last_epoch to current_iteration. builtin_warn(*args, **kwargs)
WARNING 2021-04-17T21:17:35 | py.warnings: /home/ifrahmaqsood/new-mmf/mmf-master/mmf/utils/distributed.py:272: UserWarning: 'lr_scheduler' key is not present in the checkpoint asked to be loaded. Setting lr_scheduler's last_epoch to current_iteration. builtin_warn(*args, **kwargs)
2021-04-17T21:17:35 | mmf.utils.checkpoint: Checkpoint loaded. 2021-04-17T21:17:35 | mmf.utils.checkpoint: Current num updates: 0 2021-04-17T21:17:35 | mmf.utils.checkpoint: Current iteration: 0 2021-04-17T21:17:35 | mmf.utils.checkpoint: Current epoch: 0 2021-04-17T21:17:41 | mmf.trainers.core.evaluation_loop: Starting test inference predictions 2021-04-17T21:17:41 | mmf.common.test_reporter: Predicting for vqa2 0%| | 0/13994 [00:00<?, ?it/s]
@apsdehal @vedanuj @ytsheng any updates? or can you reproduce this issue at your end?
Hi, here's what I got:
[{"test-dev": {"yes/no": 89.13, "number": 58.33, "other": 64.5, "overall": 73.94}}]
I would open up an issue on grids-feat-vqa repo to understand if you are using correct features or not. @vedanuj won't be available until May 6th, so I can't suggest if you are using correct features. The result I have provided is with the features that auto-download when you run MMF.
@apsdehal Thank you for your response. The issue is that I need to calculate the features myself. I provided the steps above that I have followed for X-152pp. If I have to train any other dataset on movie_mcan model I need to calculate the features on my own. For this, accuracy meetup is necessary with already given results.
For basic sanity check, you should first try to match the features you are extracting with the ones that are downloaded by MMF. Also, if there are minor difference in extraction, you won't be able to get same accuracy by using the original pretrained model. You will have to retrain the model from scratch.
I understand but due to resource limitations, I could not download the auto downloaded features by MMF. I'll try to make it possible to download them and will update you regarding the results.
@apsdehal reporting back. I downloaded the features and extracted the file. I checked the file sizes of both downloaded and the features that I extracted myself. Yes, you are right there is a huge file size difference between the two i-e
The test split image features
The one I calculated myself is of size 3.9M 1024.pth
The one I downloaded is of size 8.1M 1024.pth
Please let me know how the MMF features been calculated as I used the same model X-152pp for calculating features.
Also, I cross-checked my features' size extracted from https://github.com/facebookresearch/grid-feats-vqa with their pre-extracted(given) features and our size of features are same. So I think I have been calculating the features right. but unable to match MMF's
