binaural-sound-perception

binaural-sound-perception copied to clipboard

Semantic Object Prediction and Spatial Sound Super-Resolution with Binaural Sounds

Arun Balajee Vasudevan, Dengxin Dai, Luc Van Gool

This repo contains the code of our ECCV 2020 paper

License

This software is released under a creative commons license which allows for personal and research use only. You can view a license summary here.

Installation

• The code is tested with pytorch 1.5 and python 3.7.1

Data

wget http://data.vision.ee.ethz.ch/arunv/binaural_perception/data.zip
unzip data.zip 

Note: Audio track 3 (LAB-T***.WAV/LAB-T***_Tr3.WAV) and Audio track 4 (LAB-T***.WAV/LAB-T***_Tr4.WAV) of scenes from 1 to 27 are missing in the dataset. This is due to the manual error while recording the initial 27 videos of the dataset.

To extract video segments and corresponding sound time-frequency representations:

python extract_videosegments.py
python extract_spectrograms.py

Training

a) Semantic prediction, Depth prediction and Spatial sound super-resolution

python train_noBG_Paralleltask_depth.py

b) Depth prediction and Spatial sound super-resolution

python scripts/train_noBG_Paralleltask_depth_noSeman.py

c) Semantic prediction and Spatial sound super-resolution

python scripts/train_noBG_Paralleltask.py

Acknowledgement

This work is funded by Toyota Motor Europe via the research project TRACE (Toyota Research on Automated Cars in Europe) Zurich and was carried out at the CV Lab at ETH Zurich. Our codes include adapted version from the external repository of NVIDIA:

• Semantic Segmentation: https://github.com/NVIDIA/semantic-segmentation The associated license is here.

Citation

@inproceedings{vasudevan2020semantic,
  title={Semantic object prediction and spatial sound super-resolution with binaural sounds},
  author={Vasudevan, Arun Balajee and Dai, Dengxin and Van Gool, Luc},
  booktitle={European Conference on Computer Vision},
  pages={638--655},
  year={2020},
  organization={Springer}
}