Ques about the OVON task: datasets and eval

[bryce-wan]

Thank you for your outstanding work on embodied multimodal navigation!

While trying to replicate OVON related experiments, I realized that the DATA_PATH in the script did not match what was in the hm3d full dataset I downloaded, and I did not find any more information about this part of the dataset. I noticed that in this QA you mentioned that the episodes can be extracted from the GOAT-Bench dataset, so what exactly should I do to prepare this dataset?

And how can I evaluate the performance of SenseAct-NN Monolithic Policy on the OVON task? I found that there are only eval scripts for imagenav and languagenav, so I modified the config file based on the 2-goat-eval.sh accordingly and test with the objectnav dataset:

srun python -um goat_bench.run \
  --run-type eval \
  --exp-config config/experiments/ver_objectnav.yaml \
...
  habitat_baselines.rl.policy.name=GOATPolicy
...

I added habitat_baselines.rl.policy.name=GOATPolicy for SenseAct-NN Monolithic Policy and encountered an error when loading the ckpt:

  File "/ailab/user/wanchenyang/.conda/envs/goat38/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/ailab/user/wanchenyang/.conda/envs/goat38/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/ailab/user/wanchenyang/code/goat-bench/goat_bench/run.py", line 224, in <module>
    main()
  File "/ailab/user/wanchenyang/code/goat-bench/goat_bench/run.py", line 101, in main
    execute_exp(config, args.run_type)
  File "/ailab/user/wanchenyang/code/habitat-lab/habitat-baselines/habitat_baselines/run.py", line 83, in execute_exp
    trainer.eval()
  File "/ailab/user/wanchenyang/code/habitat-lab/habitat-baselines/habitat_baselines/common/base_trainer.py", line 149, in eval
    self._eval_checkpoint(
  File "/ailab/user/wanchenyang/code/goat-bench/goat_bench/trainers/ppo_trainer_no_2d.py", line 116, in _eval_checkpoint
    self.agent.load_state_dict(ckpt_dict["state_dict"], strict=False)
  File "/ailab/user/wanchenyang/.conda/envs/goat38/lib/python3.8/site-packages/torch/nn/modules/module.py", line 2041, in load_state_dict
    raise RuntimeError('Error(s) in loading state_dict for {}:\n\t{}'.format(
RuntimeError: Error(s) in loading state_dict for PPO:
	size mismatch for actor_critic.net.prev_action_embedding.weight: copying a param with shape torch.Size([8, 32]) from checkpoint, the shape in current model is torch.Size([7, 32]).
	size mismatch for actor_critic.net.state_encoder.rnn.weight_ih_l0: copying a param with shape torch.Size([2048, 1632]) from checkpoint, the shape in current model is torch.Size([2048, 640]).
	size mismatch for actor_critic.action_distribution.linear.weight: copying a param with shape torch.Size([7, 512]) from checkpoint, the shape in current model is torch.Size([6, 512]).
	size mismatch for actor_critic.action_distribution.linear.bias: copying a param with shape torch.Size([7]) from checkpoint, the shape in current model is torch.Size([6]).

Can you give me some instructions, please? Thanks in advance.

[Ram81]

@bryce-wan we have not provided separate eval scripts for individual tasks are not maintaining it actively. The above error suggests there is a mismatch in the RNN size for the policy instantiated vs the checkpoint.