TSGV-Learning-List

说明

总结了2017年至今在TSGV方向上的相关工作。 Temporal Sentence Grounding in Videos (TSGV) Natural Language Video Localization (NLVL) Video Moment Retrieval (VMR) 该任务的目标是给定一段语言描述，在一个未经裁剪的长视频中定位出该语言所描述的视频片段。

目录

• 数据集
• 相关工作
  • Survey
  • Sliding Window-based Method
  • Proposal Generated Method
  • Anchor-based Method
  • • Standard Anchor-based Method
  • • 2D-Map Anchor-based Method
  • Regression-based Method
  • Span-based Method
  • Reinforcement Leaning-based Method
  • Other Supervised Method
  • Weakly-supervised TSGV Method
  • • Multi-Instance Learning Method
  • • Reconstruction-based Method
  • • Other Weakly-supervised Method
• 参考

数据集

相关工作

Survey

• A survey of temporal activity localization via language in untrimmed videos. in ICCST 2020
• A survey on natural language video localization. in ArXiv 2021
• A survey on temporal sentence grounding in videos. in ArXiv 2021
• The Elements of Temporal Sentence Grounding in Videos: A Survey and Future Directions. in ArXiv 2022

Sliding Window-based Method

Sliding window-based method adopts a multi-scale sliding windows (SW) to generate proposal candidates.

• CTRL: Tall: Temporal activity localization via language query. in ICCV 2017. code
• MCN: Localizing moments in video with natural language. in ICCV 2017
• ROLE: Crossmodal moment localization in videos. in ACM MM 2018
• ACRN: Attentive moment retrieval in videos. in SIGIR 2018
• MAC: Mac: Mining activity concepts for language-based temporal localization. in WACV 2019. code
• MCF: Multi-modal circulant fusion for video-to-language and backward. in IJCAI 2018
• MLLC: Localizing moments in video with temporal language. in EMNLP 2018
• TCMN: Exploiting temporal relationships in video moment localization with natural language. in ACM MM 2019. code
• ASST: An attentive sequence to sequence translator for localizing video clips by natural language. in TMM 2020. code
• SLTA: Cross-modal video moment etrieval with spatial and language-temporal attention. in ICMR 2019. code
• MMRG: Multi-modal relational graph for cross-modal video moment retrieval. in CVPR 2021
• I$^2$N: Interaction-integrated network for natural language moment localization. in TIP 2021

Proposal Generated Method

Proposal generated (PG) method alleviates the computation burden of SW-based methods and generates proposals conditioned on the query.

• Text-to-Clip: Text-toclip video retrieval with early fusion and re-captioning. in ArXiv 2018
• QSPN: Multilevel language and vision integration for text-to-clip retrieval. in AAAI 2019
• SAP: Semantic proposal for activity localization in videos via sentence query. in AAAI 2019
• BPNet: Boundary proposal network for two-stage natural language video localization. in AAAI 2021
• APGN: Adaptive proposal generation network for temporal sentence localization in videos. in EMNLP 2021
• LP-Net: Natural language video localization with learnable moment proposals. in EMNLP 2021
• CMHN: Video moment localization via deep cross-modal hashing in TIP 2021.

Anchor-based Method

Anchor-based methods incorporates proposal generation into answer prediction and maintains the proposals with various learning modules.

Standard Anchor-based Method

• TGN: Temporally grounding natural sentence in video. in EMNLP 2018. code
• MAN: Man: Moment alignment network for natural language moment retrieval via iterative graph adjustmen. in CVPR 2019
• SCDM: Semantic conditioned dynamic modulation for temporal sentence grounding in videos. in NeurIPS 2019. code
• CMIN: Cross-modal interaction networks for query-based moment retrieval in videos. in SIGIR 2019. code
• SCDM$^*$: Semantic conditioned dynamic modulation for temporal sentence grounding in videos. in TPAMI 2020
• CMIN$^*$: Moment retrieval via cross-modal interaction networks with query reconstruction. in TIP 2020
• CBP: Temporally grounding language queries in videos by contextual boundary-aware prediction. in AAAI 2020. code
• FIAN: Finegrained iterative attention network for temporal language localization in videos. in ACM MM 2020
• HDRR: Hierarchical deep residual reasoning for temporal moment localization. ACM MM Asia 2021
• MIGCN: Multi-modal interaction graph convolutional network for temporal language localization in videos. in TIP 2021
• CSMGAN: Jointly cross- and self-modal graph attention network for query-based moment localization. in ACM MM 2020
• RMN: Reasoning step-by-step: Temporal sentence localization in videos via deep rectification-modulation network. in COLING 2020
• IA-Net: Progressively guide to attend: An iterative alignment framework for temporal sentence grounding. in EMNLP 2021
• DCT-Net: Dct-net: A deep co-interactive transformer network for video temporal grounding. in IVC 2021

2D-Map Anchor-based Method

• 2D-TAN: Learning 2d temporal adjacent networks formoment localization with natural language. in AAAI 2020. code
• MATN: Multi-stage aggregated transformer network for temporal language localization in videos. in CVPR 2021
• SMIN: Structured multi-level interaction network for video moment localization via language query. in CVPR 2021
• RaNet: Relation-aware video reading comprehension for temporal language grounding. in EMNLP 2021. code
• FVMR: Fast video moment retrieval. in ICCV 2021
• MS 2DTAN: Multi-scale 2d temporal adjacency networks for moment localization with natural language. in TPAMI 2021. code
• PLN: Progressive localization networks for language-based moment localization. in ArXiv 2021
• CLEAR: Coarseto-fine semantic alignment for cross-modal moment localization. in TIP 2021
• STCM-Net: Stcm-net: A symmetrical one-stage network for temporal language localization in videos. in Neurocomputing 2022
• VLG-Net: Vlg-net: Videolanguage graph matching network for video grounding. in ICCV Workshop 2021
• SV-VMR: Diving into the relations: Leveraging semantic and visual structures for video moment retrieval. in ICME 2021
• MMN: Negative sample matters: A renaissance of metric learning for temporal grounding. in AAAI 2022. code

Regression-based Method

Regression-based method computes a time pair ($t_s$, $t_e$) and compares the computed pair with ground-truth ($τ_s$, $τ_e$) for model optimization.

• ABLR: To find where you talk: Temporal sentence localization in video with attention based location regression. in AAAI 2019. code
• ExCL: ExCL: Extractive Clip Localization Using Natural Language Descriptions. in NAACL 2019
• DEBUG: DEBUG: A dense bottomup grounding approach for natural language video localization. in EMNLP 2019
• GDP: Rethinking the bottom-up framework for query-based video localization. in AAAI 2020
• CMA:: A simple yet effective method for video temporal grounding with cross-modality attention. in ArXiv 2020
• DRN: Dense regression network for video grounding. in CVPR 2020. code
• LGI: Local-global video-text interactions for temporal grounding. in CVPR 2020. code
• CPNet: Proposal-free video grounding with contextual pyramid network. in AAAI 2021
• DeNet: Embracing uncertainty: Decoupling and de-bias for robust temporal grounding. in CVPR 2021
• SSMN: Single-shot semantic matching network for moment localization in videos. in ACM TOMCCAP 2021
• HVTG: Hierarchical visual-textual graph for temporal activity localization via language. in ECCV 2020. code
• PMI: Learning modality interaction for temporal sentence localization and event captioning in videos. in ECCV 2020
• DRFT: End-to-end multi-modal video temporal grounding. in NeurIPS 2021

Span-based Method

Span-based methods aim to predict the probability of each video snippet/frame being the start and end positions of target moment.

• ExCL: ExCL: Extractive Clip Localization Using Natural Language Descriptions. in NAACL 2019
• L-Net: Localizing natural language in videos. in AAAI 2019
• VSLNet: Span-based localizing network for natural language video localization. in ACL 2020. code
• VSLNet-L$^*$: Natural language video localization: A revisit in span-based question answering framework. in TPAMI 2021
• TMLGA: Proposal-free temporal moment localization of a natural-language query in video using guided attention. in WACV 2020. code
• SeqPAN: Parallel attention network with sequence matching for video grounding. in Findings of ACL 2021. code
• CPN: Cascaded prediction network via segment tree for temporal video grounding. in CVPR 2021
• IVG: Interventional video grounding with dual contrastive learning. in CVPR 2021. code
• Local-enhanced interaction for temporal moment localization. in ICMR 2021
• CI-MHA: Cross interaction network for natural language guided video moment retrieval. in SIGIR 2021
• MQEI: Multi-level query interaction for temporal language grounding. in TITS 2021
• ACRM: Frame-wise crossmodal matching for video moment retrieval. in TMM 2021
• ABIN: Temporal textual localization in video via adversarial bi-directional interaction networks. in TMM 2021
• CSTI: Collaborative spatial-temporal interaction for language-based moment retrieval. in WCSP 2021
• DORi: Dori: Discovering object relationships for moment localization of a natural language query in a video. in WACV 2021
• CBLN: Context-aware biaffine localizing network for temporal sentence grounding. in CVPR 2021. code
• PEARL: Natural language video moment localization through query-controlled temporal convolution. in WACV 2022

Reinforcement Leaning-based Method

RL-based method formulates TSGV as a sequence decision making problem, and utilizes deep reinforcement learning techniques to solve it.

• RWM-RL: Read, watch, and move: Reinforcement learning for temporally grounding natural language descriptions in videos. in AAAI 2019
• SM-RL: Language-driven temporal activity localization: A semantic matching reinforcement learning model. in CVPR 2019
• TSP-PRL: Tree-structured policy based progressive reinforcement learning for temporally language grounding in video. in AAAI 2020
• AVMR: Adversarial video moment retrieval by jointly modeling ranking and localization. in ACM MM 2020
• STRONG: Strong: Spatio-temporal reinforcement learning for cross-modal video moment localization. in ACM MM 2020
• TripNet: Tripping through time: Efficient localization of activities in videos. in BMVC 2020
• MABAN: Maban: Multi-agent boundary-aware network for natural language moment retrieval. in TIP 2021

Other Supervised Method

• FIFO: Find and focus: Retrieve and localize video events with natural language queries. in ECCV 2018. code
• DPIN: Dual path interaction network for video moment localization. in ACM MM 2020
• Sscs: Support-set based cross-supervision for video grounding. in ICCV 2021
• DepNet: Dense events grounding in video. in AAAI 2021
• SNEAK: Sneak: Synonymous sentences-aware adversarial attack on natural language video localization. in ArXiv 2021. code
• BSP: Boundary-sensitive pre-training for temporal localization in videos. in ICCV 2021. code
• GTR: On pursuit of designing multi-modal transformer for video grounding. in EMNLP 2021. code

Weakly-supervised TSGV Method

Under weakly-supervised setting, TSGV methods only need video-query pairs but not the annotations of starting/end time.

Multi-Instance Learning Method

• TGA: Weakly supervised video moment retrieval from text queries. in CVPR 2019
• WSLLN: WSLLN:weakly supervised natural language localization networks. in EMNLP 2019
• Coarse-to-Fine: Look closer to ground better: Weakly-supervised temporal grounding of sentence in video. in ArXiv 2020
• VLANet: Vlanet: Video-language alignment network for weakly-supervised video moment retrieva. in ECCV 2020
• BAR: Reinforcement learning for weakly supervised temporal grounding of natural language in untrimmed videos. in ACM MM 2020
• CCL: Counterfactual contrastive learning for weakly-supervised vision-language grounding. in NeurIPS 2020
• AsyNCE: Asynce: Disentangling false-positives for weakly-supervised video grounding. in ACM MM 2021
• Visual co-occurrence alignment learning for weakly-supervised video moment retrieval. in ACM MM 2021
• FSAN: Fine-grained semantic alignment network for weakly supervised temporal language grounding. in Findings of EMNLP 2021
• CRM: Cross-sentence temporal and semantic relations in video activity localisation. in ICCV 2021
• LCNet: Local correspondence network for weakly supervised temporal sentence grounding. in TIP 2021
• Regularized two granularity loss function for weakly supervised video moment retrieval. in TMM 2021
• WSTAN: Weakly supervised temporal adjacent network for language grounding. in TMM 2021
• LoGAN: Logan: Latent graph co-attention network for weakly-supervised video moment retrieval. in WACV 2021

Reconstruction-based Method

• Weakly supervised dense event captioning in videos. in NeurIPS 2018
• SCN: Weakly-supervised video moment retrieval via semantic completion network. in AAAI 2020
• EC-SL: Towards bridging event captioner and sentence localizer for weakly supervised dense event captioning. in CVPR 2021
• MARN: Weakly-supervised multilevel attentional reconstruction network for grounding textual queries in videos. in ArXiv 2020
• Towards bridging video and language by caption generation and sentence localization. in ACM MM 2021

Other Weakly-supervised Method

• RTBPN: Regularized two-branch proposal networks for weakly-supervised moment retrieval in videos. in ACM MM 2020. code
• S$^4$TLG: Self-supervised learning for semi-supervised temporal language grounding. in ArXiv 2021
• PSVL: Zero-shot natural language video localization. in ICCV 2021. code
• Learning video moment retrieval without a single annotated video. in TCSVT 2021

参考

Survey of Zhang et al