habitat-lab
habitat-lab copied to clipboard
passing only episode info at every timestep during evaluation
Motivation and Context
The evaluation for ObjectNav agents takes much longer than PointNav and ImageNav. This is because, at each step, each process pipes back the episode it is evaluating on to the main process. This takes a lot of time for ObjectNav because it stores a large goal object compared to PointNav or ImageNav. This PR tries to solve this issue by only passing the required episode information to the main process. It leads to an ~8x gain in eval performance when tested with 20 parallel processes.
Related Issue: https://github.com/facebookresearch/habitat-lab/issues/597
How Has This Been Tested
I am evaluating a checkpoint on the HM3D val set. Here is the time required by the old and new code:
- Time required for evaluation with the old code: 7 hrs and 30 mins
- Time required for evaluation after bugfix: 55 mins
Types of changes
- Bug fix (non-breaking change which fixes an issue)
Checklist
- [x] My code follows the code style of this project.
- [ ] My change requires a change to the documentation.
- [ ] I have updated the documentation accordingly.
- [x] I have read the CONTRIBUTING document.
- [x] I have completed my CLA (see CONTRIBUTING)
- [ ] I have added tests to cover my changes.
- [ ] All new and existing tests passed.