EfficientViT: Multi-Scale Linear Attention for High-Resolution Dense Prediction (paper, poster)

News

If you are interested in getting updates, please join our mailing list here.

• [2024/04/23] We released the training code of EfficientViT-SAM.
• [2024/04/06] EfficientViT-SAM is accepted by eLVM@CVPR'24.
• [2024/03/19] Online demo of EfficientViT-SAM is available: https://evitsam.hanlab.ai/.
• [2024/02/07] We released EfficientViT-SAM, the first accelerated SAM model that matches/outperforms SAM-ViT-H's zero-shot performance, delivering the SOTA performance-efficiency trade-off.
• [2023/11/20] EfficientViT is available in the NVIDIA Jetson Generative AI Lab.
• [2023/09/12] EfficientViT is highlighted by MIT home page and MIT News.
• [2023/07/18] EfficientViT is accepted by ICCV 2023.

About EfficientViT Models

EfficientViT is a new family of ViT models for efficient high-resolution dense prediction vision tasks. The core building block of EfficientViT is a lightweight, multi-scale linear attention module that achieves global receptive field and multi-scale learning with only hardware-efficient operations, making EfficientViT TensorRT-friendly and suitable for GPU deployment.

Third-Party Implementation/Integration

• NVIDIA Jetson Generative AI Lab
• timm: link
• X-AnyLabeling: link

Getting Started

conda create -n efficientvit python=3.10
conda activate efficientvit
conda install -c conda-forge mpi4py openmpi
pip install -r requirements.txt

EfficientViT Applications

Segment Anything

• Datasets
• Pretrained Models
• Use in Pytorch
• Evaluation
• Visualization
• Web Demo
• Deployment using ONNX and TensorRT
• Training

Table1: Summary of All EfficientViT-SAM Variants. COCO mAP and LVIS mAP are measured using ViTDet's predicted bounding boxes as the prompt. End-to-end Jetson Orin latency and A100 throughput are measured with TensorRT and fp16.

Image Classification

• Datasets
• Pretrained Models
• Use in Pytorch
• Evaluation
• Deployment
• Training

Semantic Segmentation

• Datasets
• Pretrained Models
• Use in Pytorch
• Evaluation
• Visualization
• Deployment

Demo

• GazeSAM: Combining EfficientViT-SAM with Gaze Estimation

Contact

Han Cai: [email protected]

TODO

• [x] ImageNet Pretrained models
• [x] Segmentation Pretrained models
• [x] ImageNet training code
• [x] EfficientViT L series, designed for cloud
• [x] EfficientViT for segment anything
• [ ] EfficientViT for image generation
• [ ] EfficientViT for CLIP
• [ ] EfficientViT for super-resolution
• [ ] Segmentation training code

Citation

If EfficientViT is useful or relevant to your research, please kindly recognize our contributions by citing our paper:

@article{cai2022efficientvit,
  title={Efficientvit: Enhanced linear attention for high-resolution low-computation visual recognition},
  author={Cai, Han and Gan, Chuang and Han, Song},
  journal={arXiv preprint arXiv:2205.14756},
  year={2022}
}