visual_prompt_retrieval

visual_prompt_retrieval copied to clipboard

What Makes Good Examples for Visual In-Context Learning?

Zhang Yuanhan  Zhou Kaiyang  Liu Ziwei

S-Lab, Nanyang Technological University

TL;DR

We study on the effect of in-context examples in computer vision. We propose a Prompt Retrieval framework to automatically select examples, consisting of an unsupervised (UnsupPR) and a supervised method (SupPR).

---

[arXiv]

Updatas

[01/2023] arXiv paper has been released.

[01/2023] The code for foureground segmentation has been released.

Environment setup

conda create -n XXX python=3.8
conda activate XXX
pip install -r requirements.txt

Data preparation

Our data preparation pipeline is based on visual prompt. Please follow the dataset preparation steps for PASCAL-5i dataset in this repository.

How to run

Click the Unsup/Sup stratedgy below to see the detailed instructions on how to run the code to reproduce the results.

• Unsupervised Prompt Retrieval
• Supervised Prompt Retrieval

Performance

Here, Random is the baseline method in visual prompt, SupPR and UnsupPR are shorted for supervised prompt retrieval and unsupervised prompt retrieval respectively.

Model

The SupPR models for each pascal-5ⁱ split is uploaded in link.

Citation

If you use this code in your research, please kindly cite this work.

@inproceedings{zhang2023VisualPromptRetrieval,
      title={What Makes Good Examples for Visual In-Context Learning?}, 
      author={Yuanhan Zhang and Kaiyang Zhou and Ziwei Liu},
      year={2023},
      archivePrefix={arXiv},
}

Acknowledgments

Part of the code is borrowed from visual prompt, SupContrast, timm and mmcv.