conda create --name covr
conda activate covr

To install the necessary packages, you can use the provided requirements.txt file:

python -m pip install -r requirements.txt

Alternatively, you can manually install the following packages inside the conda environment:

python -m pip install pytorch_lightning --upgrade
python -m pip install hydra-core --upgrade
python -m pip install lightning
python -m pip install einops
python -m pip install pandas
python -m pip install opencv-python
python -m pip install timm
python -m pip install fairscale
python -m pip install tabulate
python -m pip install transformers

The code was tested on Python 3.8 and PyTorch 2.0.

WebVid-CoVR

To use the WebVid-CoVR dataset, you will have to download the WebVid videos and the WebVid-CoVR annotations.

To download the annotations, run:

bash tools/scripts/download_annotation.sh covr

To download the videos, install mpi4py and run:

python tools/scripts/download_covr.py <split>

CIRR

To use the CIRR dataset, you will have to download the CIRR images and the CIRR annotations.

To download the annotations, run:

bash tools/scripts/download_annotation.sh cirr

To download the images, follow the instructions in the CIRR repository. The default folder structure is the following:

📦 CoVR
 ┣ 📂 datasets  
 ┃ ┣ 📂 CIRR
 ┃ ┃ ┣ 📂 images
 ┃ ┃ ┃ ┣ 📂 train
 ┃ ┃ ┃ ┣ 📂 dev
 ┃ ┃ ┃ ┗ 📂 test1

FashionIQ

To use the FashionIQ dataset, you will have to download the FashionIQ images and the FashionIQ annotations.

To download the annotations, run:

bash tools/scripts/download_annotation.sh fiq

To download the images, the urls are in the FashionIQ repository. You can use the this script to download the images. Some missing images can also be found here. All the images should be placed in the same folder (datasets/fashion-iq/images).

To download the checkpoints, run:

bash tools/scripts/download_pretrained_models.sh

Before training, you will need to compute the BLIP embeddings for the videos/images. To do so, run:

# This will compute the embeddings for the WebVid-CoVR videos. 
# Note that you can use multiple GPUs with --num_shards and --shard_id
python tools/embs/save_blip_embs_vids.py --video_dir datasets/WebVid/2M/train --todo_ids annotation/webvid-covr/webvid2m-covr_train.csv 

# This will compute the embeddings for the WebVid-CoVR-Test videos.
python tools/embs/save_blip_embs_vids.py --video_dir datasets/WebVid/8M/train --todo_ids annotation/webvid-covr/webvid8m-covr_test.csv 

# This will compute the embeddings for the CIRR images.
python tools/embs/save_blip_embs_imgs.py --image_dir datasets/CIRR/images/

# This will compute the embeddings for FashionIQ images.
python tools/embs/save_blip_embs_imgs.py --image_dir datasets/fashion-iq/images/

The command to launch a training experiment is the folowing:

python train.py [OPTIONS]

The parsing is done by using the powerful Hydra library. You can override anything in the configuration by passing arguments like foo=value or foo.bar=value. See Options parameters section at the end of this README for more details.

The command to evaluate is the folowing:

python test.py test=<test> [OPTIONS]

Datasets:

• data=webvid-covr: WebVid-CoVR datasets.
• data=cirr: CIRR dataset.
• data=fashioniq: FashionIQ dataset.

Tests:

• test=all: Test on WebVid-CoVR, CIRR and all three Fashion-IQ test sets.
• test=webvid-covr: Test on WebVid-CoVR.
• test=cirr: Test on CIRR.
• test=fashioniq: Test on all three Fashion-IQ test sets (dress, shirt and toptee).

Checkpoints:

• model/ckpt=blip-l-coco: Default checkpoint for BLIP-L finetuned on COCO.
• model/ckpt=webvid-covr: Default checkpoint for CoVR finetuned on WebVid-CoVR.
• model/ckpt=fashioniq-all-ft_covr: Default checkpoint pretrained on WebVid-CoVR and finetuned on FashionIQ.
• model/ckpt=cirr_ft-covr+gt: Default checkpoint pretrained on WebVid-CoVR and finetuned on CIRR.

Training

• trainer=gpu: training with CUDA, change devices to the number of GPUs you want to use.
• trainer=ddp: training with Distributed Data Parallel (DDP), change devices and num_nodes to the number of GPUs and number of nodes you want to use.
• trainer=cpu: training on the CPU (not recommended).

Logging

• trainer/logger=csv: log the results in a csv file. Very basic functionality.
• trainer/logger=wandb: log the results in wandb. This requires to install wandb and to set up your wandb account. This is what we used to log our experiments.
• trainer/logger=<other>: Other loggers (not tested).

Machine

• machine=server: You can change the default path to the dataset folder and the batch size. You can create your own machine configuration by adding a new file in configs/machine.

Experiment

There are many pre-defined experiments from the paper in configs/experiments. Simply add experiment=<experiment> to the command line to use them.