WALT:Watch and Learn 2D Amodal Representation using time-lapse imagery

N Dinesh Reddy, Robert Tamburo, Srinivasa G. Narasimhan

IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.

[Project] [Paper] [Supp][Dataset] [Bibtex ]

Installation

Setting up with docker

All the stable releases of docker-ce installed from https://docs.docker.com/install/

Setting up the docker

docker build -t walt docker/

Implementation of WALT

We Will show the steps to follow to train the walt network to produce amodal segmentation results on any camera in the wild.

Generating CWALT dataset

Firstly you need to generate the CWALT data composition. To do that we need to download the walt dataset from HERE.

The final folder format to train on carfusion data needs to look :

WALT
 └─data
     └─cam2
         └─cam2.json
         └─week1
             2021-05-01T00-02-23.775683.jpg
	      2021-05-01T00-44-55.207427.jpg
             ...
         └─week2
             2021-05-08T00-00-59.416878.jpg
	      2021-05-08T00-18-03.210882.jpg
             ...
          ...

         └─T01-median_image.jpg
           T02-median_image.jpg
           T03-median_image.jpg
           ...	
               

Then CWALT dataset can be generated using

docker run --gpus all --shm-size=8g -v $PWD:/code walt python cwalt_generate.py

Training

For Training the WALNET for the data use the following script:

sudo docker run --gpus all --shm-size=24g -v $PWD:/code walt python train.py configs/walt/walt_vehicle.py

Testing

For Evaluations of the trained model run the following commands

sudo docker run --gpus all --shm-size=24g -v $PWD:/code walt python test.py configs/walt/walt_vehicle.py {Trained Model}.pth --eval bbox

Inference Script on Demo Images

Inference on the images with occlusions run

mkdir data/models
wget https://www.cs.cmu.edu/~walt/models/walt_people.pth -O data/models/walt_people.pth
wget https://www.cs.cmu.edu/~walt/models/walt_vehicle.pth -O data/models/walt_vehicle.pth
sudo docker run --gpus all --shm-size=24g -v $PWD:/code walt python infer.py 

Demo of Amodal Segmentation

Using Gradio we created a very simple cpu dependent demo of our network which can be run on any computer.

virtualenv waltdemo -p python3
source waltdemo/bin/activate
pip install -r requirements.txt
python3 app.py

We have a live demo online at [CVPR WALT DEMO] and [WALT DEMO]

Citation

@InProceedings{Reddy_2022_CVPR,
    author    = {Reddy, N. Dinesh and Tamburo, Robert and Narasimhan, Srinivasa G.},
    title     = {WALT: Watch and Learn 2D Amodal Representation From Time-Lapse Imagery},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2022},
    pages     = {9356-9366}
}