VXN-Dataset

The official repository of the NeurIPS 2022 paper: Towards Versatile Embodied Navigation. VXN is the abbreviation of Vision-$X$ Navigation, a large-scale test bed for multi-task embodied navigation.

Hanqing Wang | Wei Liang | Luc Van Gool | Wenguan Wang

Environment Installation

Create a python environment and install the required packages using the following scripts:

conda create -n vxn --file requirements.txt
conda activate vxn

Dataset

Create the folder for datasets using the following scripts:

mkdir data

Matterport3D Scenes

Download the matterport3D dataset following the instruction here.

Navigation Tasks

Download datasets for image-goal nav., audio-goal nav., object-goal nav., and vision-language nav. tasks, and uncompressed it under the path data/datasets.

Continuous Audio Rendering

1. Download the rendered BRIRs (binaural room impulse responses) (887G) for Matterport3D scenes here. Put data.mdb under the path data/binaural_rirs_lmdb/.
2. Download the alignment data (505G) for discrete BRIRs here. Put data.mdb under the path data/align_lmdb/.

Training

For a multi-node cluster, run the following script to start the training.

bash sbatch_scripts/sub_job.sh

Evaluation

Run the following script to evaluate the trained model for each task.

bash sbatch_scripts/eval_mt.sh

TODOs

1. [ ] Release the full dataset.
2. [ ] Release the checkpoints.

Citation

If you find our project useful, please consider citing us:

@inproceedings{wang2022towards,
  title = {Towards Versatile Embodied Navigation},
  author = {Wang, Hanqing and Liang, Wei and Van Gool, Luc and Wang, Wenguan},
  booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
  year = {2022}
}

License

The VXN codebase is MIT licensed. Trained models and task datasets are considered data derived from the mp3d scene dataset. Matterport3D based task datasets and trained models are distributed with Matterport3D Terms of Use and under CC BY-NC-SA 3.0 US license.

Acknowledgment

This repository is built upon the following publicly released projects:

• habitat-lab
• sound-spaces
• VLN-CE

Thanks to the authors who create those great prior works.