Depth agents do not outperform RGB agents on "test scenes data"

[at82]

I have trained RGB and Depth agents for pointnav task  using "ppo_pointnav_example.yaml " and evaluating them using the same yaml script. Contrary to the observation in the Habitat paper, where the Depth agent outperforms the RGB one, my results show that on the "test scenes data", the average 'SPL' score over the evaluation episodes are almost the same for both agents: Depth agent: 0.78   RGBa gent: 0.76

The difference reported in the paper is one order of magnitude larger than the difference I'm seeing here. I was wondering if there were a good explanation for this?

[mathfac]

Thank you for the question. To reproduce the ICCV19 results it is advised to use related config https://github.com/facebookresearch/habitat-lab/blob/master/habitat_baselines/config/pointnav/ppo_pointnav_habitat_iccv19.yaml

As you’re using the example config that has same one train and test scene as I remember, the results can be just the evidence of overfititing.