Codes for "Learning to Steer by Mimicking Features from Heterogeneous Auxiliary Networks".

Besides, our project page is now available at FM-Net.

Demo video

• Performance of auxiliary networks on unseen target data:

• Performance of FM-Net:

Content:

• Installation
• Datasets
  • Udacity
  • Comma-ai
  • BDD100K
• Semantic-Segmentation
• Steering-Control
  • Test
  • Train
• Performance
• Others
  • Citation
  • Acknowledgement
  • Contact

Installations

conda create -n tensorflow_gpu pip python=2.7
source activate tensorflow_gpu
pip install --upgrade tensorflow-gpu==1.4
conda install pytorch torchvision -c pytorch

Datasets

Udacity

The whole dataset is available at Udacity.

Comma-ai

The whole dataset is available at Comma-ai.

BDD100K

The whole dataset is available at BDD100K.

Semantic-Segmentation

FCN (mIoU 71.03%)

cd semantic-segmentation
python3 main.py VOCAug FCN train val --lr 0.01 --gpus 0 1 2 3 4 5 6 7 --npb

PSPNet

python3 train_pspnet.py VOCAug PSPNet train val --lr 0.01 --gpus 0 1 2 3 4 5 6 7 --npb --test_size 473

Note that you can use the code to train models (e.g., PSPNet, SegNet and FCN) in Cityscape.

Steering-Control

Test

cd steering-control
CUDA_VISIBLE_DEVICES="0" python 3d_resnet_lstm.py

Note that you need to read 3d_resnet_lstm.py and options.py carefully and modify the path accordingly. Note that current setting is used for Udacity dataset. To run the codes for Comma.ai dataset, please refer to Comma-ai and our paper to modify several parameters.

Train

CUDA_VISIBLE_DEVICES="0" python 3d_resnet_lstm.py --flag train

Note that the ImageNet pre-trained model is available here.

Performance

1. Udacity testing set:

2. Comma-ai testing set:

3. BDD100K testing set:

Others

Citation

If you use the codes, please cite the following publications:

@article{hou2018learning,
  title={Learning to Steer by Mimicking Features from Heterogeneous Auxiliary Networks},
  author={Hou, Yuenan and Ma, Zheng and Liu, Chunxiao and Loy, Chen Change},
  journal={arXiv preprint arXiv:1811.02759},
  year={2018}
}

Acknowledgement

This repo is built upon Udacity.

Contact

If you have any problems in reproducing the results, just raise an issue in this repo.

To-Do List:

• [x] Release codes for steering control

• [x] Attach original experimental results

• [x] Clean all codes, make them readable and reproducable

• [ ] Release codes for BDD100K dataset