oc-fewshot-public

Code for our ICLR 2021 paper Wandering Within a World: Online Contextualized Few-Shot Learning [arxiv]

RoamingRooms Dataset

Although our code base is MIT licensed, the RoamingRooms dataset is not since it is derived from the Matterport3D dataset.

To download the RoamingRooms dataset, you first need to sign the agreement for a non-commercial license here.

Then, you need to submit a request here. We will manually approve your request afterwards.

For inquiries, please email: [email protected]

The whole dataset is around 60 GB. It has 1.2M video frames with 7k unique object instance classes. Please refer to our paper for more statistics of the dataset.

System Requirements

Our code is tested on Ubuntu 18.04 with GPU capability. We provide docker files for reproducible environments. We recommend at least 20GB CPU memory and 11GB GPU memory. 2-4 GPUs are required for multi-GPU experiments. Our code is based on TensorFlow 2.

Installation Using Docker (Recommended)

1. Install protoc from here.

2. Run make to build proto buffer configuration files.

3. Install docker and nvidia-docker.

4. Build the docker container using ./build_docker.sh.

5. Modify the environment paths. You need to change DATA_DIR and OURPUT_DIR in setup_environ.sh. DATA_DIR is the main folder where datasets are placed and OUTPUT_DIR is the main folder where training models are saved.

Installation Using Conda

1. Install protoc from here.

2. Run make to build proto buffer configuration files.

3. Modify the environment paths. You need to change DATA_DIR and OURPUT_DIR in setup_environ.sh. DATA_DIR is the main folder where datasets are placed and OUTPUT_DIR is the main folder where training models are saved.

4. Create a conda environment:

conda create -n oc-fewshot python=3.6
conda activate oc-fewshot
conda install pip

5. Install CUDA 10.1

6. Install OpenMPI 4.0.0

7. Install NCCL 2.6.4 for CUDA 10.1

8. Modify installation paths in install.sh

9. Run install.sh

Setup Datasets

1. To set up the Omniglot dataset, run script/download_omniglot.sh. This script will download the Omniglot dataset to DATA_DIR.

2. To set up the Uppsala texture dataset (for spatiotemporal cue experiments), run script/download_uppsala.sh. This script will download the Uppsala texture dataset to DATA_DIR.

RoamingOmniglot Experiments

To run training on your own, use the following command.

./run_docker.sh {GPU_ID} python -m fewshot.experiments.oc_fewshot \
  --config {MODEL_CONFIG_PROTOTXT} \
  --data {EPISODE_CONFIG_PROTOTXT} \
  --env configs/environ/roaming-omniglot-docker.prototxt \
  --tag {TAG} \
  [--eval]

• MODEL_CONFIG_PROTOTXT can be found in configs/models.
• EPISODE_CONIFG_PROTOTXT can be found in configs/episodes.
• TAG is the name of the saved checkpoint folder.
• When the model finishes training, add the --eval flag to evaluate.

For example, to train CPM on the semisupervised benchmark:

./run_docker.sh 0 python -m fewshot.experiments.oc_fewshot \
  --config configs/models/roaming-omniglot/cpm.prototxt \
  --data configs/episodes/roaming-omniglot/roaming-omniglot-150-ssl.prototxt \
  --env configs/environ/roaming-omniglot-docker.prototxt \
  --tag roaming-omniglot-ssl-cpm

All of our code is tested using GTX 1080 Ti with 11GB GPU memory. Note that the above command uses a single GPU. Our original experiments in the paper is performed using two GPUs, with twice the batch size and doubled learning rate. To run that setting, use the following command:

./run_docker_hvd_01.sh python -m fewshot.experiments.oc_fewshot_hvd \
  --config {MODEL_CONFIG_PROTOTXT} \
  --data {EPISODE_CONFIG_PROTOTXT} \
  --env configs/environ/roaming-omniglot-docker.prototxt \
  --tag {TAG}

RoamingRooms Experiments

Below we include command to run experiments on RoamingRooms. Our original experiments in the paper is performed using four GPUs, with batch size to be 8. To run that setting, use the following command:

./run_docker_hvd_0123.sh python -m fewshot.experiments.oc_fewshot_hvd \
  --config {MODEL_CONFIG_PROTOTXT} \
  --data {EPISODE_CONFIG_PROTOTXT} \
  --env configs/environ/roaming-rooms-docker.prototxt \
  --tag {TAG}

When evaluate, use --eval --usebest to pick the checkpoint with the highest validation performance.

Results

Table 1: RoamingOmniglot Results (Supervised)

Table 2: RoamingOmniglot Results (Semi-supervised)

Table 3: RoamingRooms Results (Supervised)

Table 4: RoamingRooms Results (Semi-supervised)

Table 5: RoamingImageNet Results (Supervised)

* denotes using pretrained CNN.

Table 6: RoamingImageNet Results (Semi-supervised)

* denotes using pretrained CNN.

To-Do

• Add a data iterator based on PyTorch (contribution welcome).

Citation

If you use our code, please consider cite the following:

• Mengye Ren, Michael L. Iuzzolino, Michael C. Mozer and Richard S. Zemel. Wandering Within a World: Online Contextualized Few-Shot Learning. In ICLR, 2021.

@inproceedings{ren21ocfewshot,
  author    = {Mengye Ren and
               Michael L. Iuzzolino and
               Michael C. Mozer and
               Richard S. Zemel},
  title     = {Wandering Within a World: Online Contextualized Few-Shot Learning},
  booktitle = {9th International Conference on Learning Representations, {ICLR}},
  year      = {2021}
}