Awesome Video Object Segmentation

Recent Advances in Video Object Segmentation (VOS). VOS works before 2022 can be found in our review paper:

Deep Learning for Video Object Segmentation: A Review / paper / project page

:teddy_bear: We indicate different VOS types with coloured squares:

:blue_square: SVOS: Semi-Supervised VOS (also termed as One-Shot VOS)

:green_square: UVOS: Un-Supervised VOS (also termed as Zero-Shot VOS)

:orange_square: RVOS: Referring VOS (also termed as Language-Guided VOS)

:red_square: AVOS: Audio-guided VOS (also termed as Audio-Visual Video Segmentation)

:white_large_square: XVOS: Other types of VOS

:teddy_bear: Please feel free to send us pull requests to add VOS works.

Links for a quick jump: ArXiv 2024, 🔥CVPR 2024🔥, AAAI 2024, EMNLP 2023, NeurIPS 2023, ACMMM 2023, ICCV 2023, CVPR 2023, IJCAI 2023, AAAI 2023, Journals 2023, Earlier ArXiv 2023, NeurIPS 2022, ECCV 2022, CVPR 2022, AAAI 2022, Journals 2022

🏁 Active Workshops/Challenges

:blue_square: SVOS    -    VOTS 2024

:blue_square: SVOS :orange_square: RVOS    -    PVUW 2024

ArXiv 2024 (Within the last 6 months)

:blue_square: SVOS Apr    -    paper / code    -    Spatial-Temporal Multi-level Association for Video Object Segmentation

:orange_square: RVOS Apr    -    paper / code    -    Decoupling Static and Hierarchical Motion Perception for Referring Video Segmentation

:orange_square: RVOS Mar    -    paper / code    -    Towards Temporally Consistent Referring Video Object Segmentation

:blue_square: SVOS Mar    -    paper / code    -    Efficient Video Object Segmentation via Modulated Cross-Attention Memory

:orange_square: RVOS :red_square: AVOS Mar    -    paper / code & dataset    -    R2-Bench: Benchmarking the Robustness of Referring Perception Models under Perturbations

:orange_square: RVOS Mar    -    paper / code    -    Exploring Pre-trained Text-to-Video Diffusion Models for Referring Video Object Segmentation

:blue_square: SVOS Mar    -    paper / code    -    OneVOS: Unifying Video Object Segmentation with All-in-One Transformer Framework

⬜ XVOS Mar    -    paper / code    -    ClickVOS: Click Video Object Segmentation

:red_square: AVOS Feb    -    paper / code    -    Bootstrapping Audio-Visual Segmentation by Strengthening Audio Cues

:white_large_square: XVOS Jan    -    paper / code    -    Vivim: a Video Vision Mamba for Medical Video Object Segmentation

:white_large_square: XVOS Jan    -    paper / code    -    Self-supervised Video Object Segmentation with Distillation Learning of Deformable Attention

:white_large_square: XVOS Jan    -    paper / code    -    Explore Synergistic Interaction Across Frames for Interactive Video Object Segmentation

:orange_square: RVOS Dec    -    paper / code    -    Tracking with Human-Intent Reasoning

:blue_square: SVOS :orange_square: RVOS Dec    -    paper / code    -    UniRef++: Segment Every Reference Object in Spatial and Temporal Spaces

:green_square: UVOS Dec    -    paper / code    -    Appearance-based Refinement for Object-Centric Motion Segmentation

:blue_square: SVOS :orange_square: RVOS Dec    -    paper / code    -    General Object Foundation Model for Images and Videos at Scale

:blue_square: SVOS Dec    -    paper / code    -    M3T: Multi-Scale Memory Matching for Video Object Segmentation and Tracking

:red_square: AVOS Dec    -    paper / code    -    Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation

:orange_square: RVOS Nov    -    paper / code    -    VIDiff: Translating Videos via Multi-Modal Instructions with Diffusion Models

:green_square: UVOS Nov    -    paper / code    -    Betrayed by Attention: A Simple yet Effective Approach for Self-supervised Video Object Segmentation

:white_large_square: XVOS Nov    -    paper / code    -    Sketch-based Video Object Segmentation: Benchmark and Analysis

:white_large_square: XVOS Nov    -    paper / code    -    Learning the What and How of Annotation in Video Object Segmentation

CVPR 2024

:white_large_square: XVOS    -    paper / code    -    Point-VOS: Pointing Up Video Object Segmentation

:red_square: AVOS    -    paper / code    -    Towards Robust Audiovisual Segmentation in Complex Environments with Quantization-based Semantic Decomposition

:red_square: AVOS    -    paper / code    -    Cooperation Does Matter: Exploring Multi-Order Bilateral Relations for Audio-Visual Segmentation

:red_square: AVOS    -    paper / code    -    A Closer Look at Audio-Visual Segmentation

:green_square: UVOS    -    paper / code    -    Depth-aware Test-Time Training for Zero-shot Video Object Segmentation

:green_square: UVOS    -    paper / code    -    Dual Prototype Attention for Unsupervised Video Object Segmentation

:green_square: UVOS    -    paper / code    -    Guided Slot Attention for Unsupervised Video Object Segmentation

:orange_square: RVOS    -    paper / code    -    LoSh: Long-Short Text Joint Prediction Network for Referring Video Object Segmentation

:orange_square: RVOS Dec    -    paper / code    -    Universal Segmentation at Arbitrary Granularity with Language Instruction

:blue_square: SVOS :orange_square: RVOS    -    paper / code    -    UniVS: Unified and Universal Video Segmentation with Prompts as Queries

:blue_square: SVOS    -    paper / code    -    Event-assisted Low-Light Video Object Segmentation

:blue_square: SVOS    -    paper / code    -    Putting the Object Back into Video Object Segmentation

AAAI 2024

:orange_square: RVOS :red_square: AVOS    -    paper / code    -    Referred by Multi-Modality: A Unified Temporal Transformer for Video Object Segmentation

:green_square: UVOS    -    paper / code    -    Generalizable Fourier Augmentation for Unsupervised Video Object Segmentation

EMNLP 2023

:orange_square: RVOS    -    paper / code    -    Towards Noise-Tolerant Speech-Referring Video Object Segmentation: Bridging Speech and Text (Spoken language as referring guidance)

NeurIPS 2023

:orange_square: RVOS    -    paper / code    -    SOC: Semantic-Assisted Object Cluster for Referring Video Object Segmentation

:blue_square: SVOS    -    paper / code    -    Test-time Training for Matching-based Video Object Segmentation

:blue_square: SVOS    -    paper / code    -    From ViT Features to Training-free Video Object Segmentation via Streaming-data Mixture Models

ACM MM 2023

:green_square: UVOS    -    paper / code    -    SimulFlow: Simultaneously Extracting Feature and Identifying Target for Unsupervised Video Object Segmentation

:green_square: UVOS    -    paper / code    -    Temporally Efficient Gabor Transformer for Unsupervised Video Object Segmentation

:blue_square: SVOS    -    paper / code    -    Exploring the Adversarial Robustness of Video Object Segmentation via One-shot Adversarial Attacks

:red_square: AVOS    -    paper / code    -    CATR: Combinatorial-Dependence Audio-Queried Transformer for Audio-Visual Video Segmentation

:red_square: AVOS    -    paper / code    -    Audio-Visual Segmentation by Exploring Cross-Modal Mutual Semantics

ICCV 2023

:white_large_square: XVOS    -    paper / code    -    Multi-grained Temporal Prototype Learning for Few-shot Video Object Segmentation

:green_square: UVOS    -    paper / code    -    Time Does Tell: Self-Supervised Time-Tuning of Dense Image Representations (self-supervised learning for UVOS)

:green_square: UVOS    -    paper / code    -    Isomer: Isomerous Transformer for Zero-Shot Video Object Segmentation

:green_square: UVOS    -    paper / code    -    Unsupervised Video Object Segmentation with Online Adversarial Self-Tuning

:green_square: UVOS :orange_square: RVOS    -    paper / code    -    DEVA: Tracking Anything with Decoupled Video Segmentation

:orange_square: RVOS    -    paper / code    -    Temporal Collection and Distribution for Referring Video Object Segmentation

:orange_square: RVOS    -    paper / code    -    Robust Referring Video Object Segmentation with Cyclic Structural Consensus

:orange_square: RVOS    -    paper / code    -    Spectrum-guided Multi-granularity Referring Video Object Segmentation

:orange_square: RVOS    -    paper / code    -    OnlineRefer: A Simple Online Baseline for Referring Video Object Segmentation

:orange_square: RVOS    -    paper / code    -    Learning Cross-Modal Affinity for Referring Video Object Segmentation Targeting Limited Samples

:orange_square: RVOS    -    paper / code    -    HTML: Hybrid Temporal-scale Multimodal Learning Framework for Referring Video Object Segmentation

:orange_square: RVOS    -    paper / code & dataset    -    MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions

:blue_square: SVOS    -    paper / code    -    Integrating Boxes and Masks: A Multi-Object Framework for Unified Visual Tracking and Segmentation

:blue_square: SVOS    -    paper / code    -    XMem++: Production-level Video Segmentation From Few Annotated Frames

:blue_square: SVOS    -    paper / code    -    Scalable Video Object Segmentation with Simplified Framework

:blue_square: SVOS    -    paper / code    -    Alignment Before Aggregation: Trajectory Memory Retrieval Network for Video Object Segmentation

:blue_square: SVOS    -    paper / code    -    SegGPT: Segmenting Everything In Context

:blue_square: SVOS    -    paper / code & dataset    -    LVOS: A Benchmark for Long-term Video Object Segmentation

:blue_square: SVOS    -    paper / code & dataset    -    MOSE: A New Dataset for Video Object Segmentation in Complex Scenes

CVPR 2023

:green_square: UVOS    -    paper / code    -    MED-VT: Multiscale Encoder-Decoder Video Transformer with Application to Object Segmentation

:blue_square: SVOS    -    paper / code    -    Boosting Video Object Segmentation via Space-time Correspondence Learning

:blue_square: SVOS :orange_square: RVOS    -    paper / code    -    Universal Instance Perception as Object Discovery and Retrieval

:blue_square: SVOS    -    paper / code    -    TarViS: A Unified Approach for Target-Based Video Segmentation

:blue_square: SVOS    -    paper / code    -    Two-shot Video Object Segmetnation

:blue_square: SVOS    -    paper / code    -    MobileVOS: Real-Time Video Object Segmentation Contrastive Learning meets Knowledge Distillation

:blue_square: SVOS    -    paper / code    -    Look Before You Match: Instance Understanding Matters in Video Object Segmentation

:white_large_square: XVOS    -    paper / code & dataset    -    Breaking the “Object” in Video Object Segmentation

IJCAI 2023

:red_square: AVOS    -    paper / code    -    Discovering Sounding Objects by Audio Queries for Audio Visual Segmentation

:blue_square: SVOS    -    paper / code & dataset    -    Video Object Segmentation in Panoptic Wild Scenes

AAAI 2023

:blue_square: SVOS    -    paper / code    -    Learning to Learn Better for Video Object Segmentation

Journals 2023

:green_square: UVOS    -    paper / code    -    TIP Hierarchical Graph Pattern Understanding for Zero-Shot Video Object Segmentation

:green_square: UVOS    -    paper / code    -    TCSVT Online Unsupervised Video Object Segmentation via Contrastive Motion Clustering

:blue_square: SVOS    -    paper / code    -    TIP Hierarchical Co-Attention Propagation Network for Zero-Shot Video Object Segmentation

:orange_square: RVOS    -    paper / code    -    TPAMI VLT: Vision-Language Transformer and Query Generation for Referring Segmentation

:orange_square: RVOS    -    paper / code    -    TPAMI Local-Global Context Aware Transformer for Language-Guided Video Segmentation

Earlier Arxiv 2023

:blue_square: SVOS    -    paper / code    -    Sub-token ViT Embedding via Stochastic Resonance Transformers (support svos)

:blue_square: SVOS    -    paper / code & dataset    -    PanoVOS: Bridging Non-panoramic and Panoramic Views with Transformer for Video Segmentation

:green_square: UVOS    -    paper / code    -    Treating Motion as Option with Output Selection for Unsupervised Video Object Segmentation

:red_square: AVOS    -    paper / code    -    Rethinking Audiovisual Segmentation with Semantic Quantization and Decomposition

:blue_square: SVOS    -    paper / code    -    Joint Modeling of Feature, Correspondence, and a Compressed Memory for Video Object Segmentation

:orange_square: RVOS :red_square: AVOS    -    paper / code    -    EPCFormer: Expression Prompt Collaboration Transformer for Universal Referring Video Object Segmentation

:orange_square: RVOS    -    paper / code    -    Learning Referring Video Object Segmentation from Weak Annotation

:green_square: UVOS    -    paper / code    -    UVOSAM: A Mask-free Paradigm for Unsupervised Video Object Segmentation via Segment Anything Model

:white_large_square: XVOS    -    paper / code    -    Segment and Track Anything

:white_large_square: XVOS    -    paper / code    -    Track Anything: Segment Anything Meets Videos

:white_large_square: XVOS    -    paper / code    -    Reliability-Hierarchical Memory Network for Scribble-Supervised Video Object Segmentation

:blue_square: SVOS    -    paper / code    -    Tracking Anything in High Quality

:orange_square: RVOS    -    paper / code    -    Referring Video Object Segmentation with Inter-Frame Interaction and Cross-Modal Correlation

:orange_square: RVOS    -    paper / code    -    RefSAM: Efficiently Adapting Segmenting Anything Model for Referring Video Object Segmentation

:white_large_square: XVOS    -    paper / code    -    Segment Anything Meets Point Tracking

NeurIPS 2022

:blue_square: SVOS    -    paper / code    -    Decoupling Features in Hierarchical Propagation for Video Object Segmentation

:white_large_square: XVOS    -    paper / code    -    Self-supervised Amodal Video Object Segmentation

ECCV 2022

:blue_square: SVOS    -    paper / code    -    XMem: Long-Term Video Object Segmentation with an Atkinson-Shiffrin Memory Model

:blue_square: SVOS    -    paper / code    -    BATMAN: Bilateral Attention Transformer in Motion-Appearance Neighboring Space for Video Object Segmentation

:blue_square: SVOS    -    paper / code    -    Learning Quality-aware Dynamic Memory for Video Object Segmentation

:blue_square: SVOS    -    paper / code    -    Tackling Background Distraction in Video Object Segmentation

:blue_square: SVOS    -    paper / code    -    Global Spectral Filter Memory Network for Video Object Segmentation

:green_square: UVOS    -    paper / code    -    Hierarchical Feature Alignment Network for Unsupervised Video Object Segmentation

CVPR 2022

:orange_square: RVOS    -    paper / code    -    End-to-End Referring Video Object Segmentation With Multimodal Transformers

:orange_square: RVOS    -    paper / code    -    Language As Queries for Referring Video Object Segmentation

:orange_square: RVOS    -    paper / code    -    Language-Bridged Spatial-Temporal Interaction for Referring Video Object Segmentation

:orange_square: RVOS    -    paper / code    -    Multi-Level Representation Learning With Semantic Alignment for Referring Video Object Segmentation

:blue_square: SVOS    -    paper / code    -    Recurrent Dynamic Embedding for Video Object Segmentation

:blue_square: SVOS    -    paper / code    -    Accelerating Video Object Segmentation With Compressed Video

:blue_square: SVOS    -    paper / code    -    SWEM: Towards Real-Time Video Object Segmentation With Sequential Weighted Expectation-Maximization

:blue_square: SVOS    -    paper / code    -    Per-Clip Video Object Segmentation

:white_large_square: XVOS    -    paper / code    -    Wnet: Audio-Guided Video Object Segmentation via Wavelet-Based Cross-Modal Denoising Networks

:white_large_square: XVOS    -    paper / code & dataset    -    YouMVOS: An Actor-Centric Multi-Shot Video Object Segmentation Dataset

AAAI 2022

:blue_square: SVOS    -    paper / code    -    Siamese Network with Interactive Transformer for Video Object Segmentation

:blue_square: SVOS    -    paper / code    -    Reliable Propagation-Correction Modulation for Video Object Segmentation

:orange_square: RVOS    -    paper / code    -    You Only Infer Once: Cross-Modal Meta-Transfer for Referring Video Object Segmentation

:green_square: UVOS    -    paper / code    -    Iteratively Selecting an Easy Reference Frame Makes Unsupervised Video Object Segmentation Easier

Journals 2022

:blue_square: SVOS    -    paper / code    -    TPAMI Video Object Segmentation Using Kernelized Memory Network With Multiple Kernels

:blue_square: SVOS    -    paper / code    -    TIP From Pixels to Semantics: Self-Supervised Video Object Segmentation With Multiperspective Feature Mining

:blue_square: SVOS    -    paper / code    -    TIP Delving Deeper Into Mask Utilization in Video Object Segmentation

:blue_square: SVOS    -    paper / code    -    TIP Adaptive Online Mutual Learning Bi-Decoders for Video Object Segmentation

End of the list. :seedling:

VOS papers and datasets before 2022 could be found below:

Deep Learning for Video Object Segmentation: A Review / paper / project page