AOT: Associating Objects with Transformers for Video Object Segmentation

This project is used to update news related to AOT (NeurIPS 2021), a highly effective and efficient VOS (video object segmentation) framework.

Implementations and Results

The implementations of AOT can be found below:

1. PyTorch

   • AOT-Benchmark from ZJU. Thanks for such an excellent implementation.

2. PaddlePaddle

   We are preparing an official PaddlePaddle implementation.

News

• 2022/03: An extension of AOT, AOST (under review), is available now. AOST is a more robust and flexible framework, supporting run-time speed-accuracy trade-offs.

• 2021/10: The conference paper has been accepted by NeurIPS 2021 (score 8/8/7/8, OpenReview).

• 2021/05: WINNER - We ranked 1st in the Track 1 (Video Object Segmentation) of the 3rd Large-scale Video Object Segmentation Challenge.

About AOT

In AOT, we propose an identification mechanism, which enables us to model, propogate, and segment multiple objects as efficiently as processing a single object. Based on the identification mechanism, the AOT framework is lightweight (with less than 10M parameters in default) yet powerful (achieving SOTA performance). Besides, we propose Long Short-Term Transformer (LSTT) for propogating temporal information hierachically, and the balance of performance and efficiency is convenient by adding or reducing LSTT blocks now in VOS.

Citations

Please consider citing the related paper(s) in your publications if it helps your research.

@article{yang2021aost,
  title={Associating Objects with Scalable Transformers for Video Object Segmentation},
  author={Yang, Zongxin and Miao, Jiaxu and Wang, Xiaohan and Wei, Yunchao and Yang, Yi},
  journal={arXiv preprint arXiv:2203.11442},
  year={2022}
}
@inproceedings{yang2021aot,
  title={Associating Objects with Transformers for Video Object Segmentation},
  author={Yang, Zongxin and Wei, Yunchao and Yang, Yi},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2021}
}