HOI-Learning-List

Some recent (2015-now) Human-Object Interaction Learing studies. If you find any errors or problems, please feel free to comment.

A list of Transfomer-based vision works: https://github.com/DirtyHarryLYL/Transformer-in-Vision.

Dataset/Benchmark

• Bongard-HOI [Paper], [Code]

• SWiG-HOI, [Paper], [Website]

• New Metric: mPD, [Paper], [Code]

• DIABOLO [Paper], [Website]

• HOI-COCO (CVPR2021) [Website]

• PaStaNet-HOI (TPAMI2021) [Benchmark]

• HAKE (CVPR2020) [YouTube] [bilibili] [Website] [Paper] [HAKE-Action-Torch] [HAKE-Action-TF]

• Ambiguous-HOI (CVPR2020) [Website] [Paper]

• HICO-DET (WACV2018) [Website] [Paper]

• HCVRD (AAAI2018) [Website] [Paper]

• V-COCO (May 2015) [Website] [Paper]

• HICO (ICCV2015) [Website] [Paper]

• OpenImage [Website] [Paper]

• PIC [Website]

More...

Video HOI Datasets

• VidHOI [Paper]

• AVA [Website], HOIs (human-object, human-human) and pose (body motion) actions

• Action Genome [Website], action verbs and spatial relationships

• CAD120 [Paper], [Website]

• Sth-else [Paper], [Website]

Method

HOI Image Generation

• Exploiting Relationship for Complex-scene Image Generation (arXiv 2021.04) [Paper]

• Specifying Object Attributes and Relations in Interactive Scene Generation (arXiv 2019.11) [Paper]

HOI Recognition: Image-based, to recognize all the HOIs in one image.

• RelViT (ICLR'22) [Paper], [Code]

• DEFR (arXiv 2021.12) [Paper]

• Interaction Compass (ICCV 2021) [Paper]

• DEFR-CLIP (arXiv 2021.07) [Paper]

• PaStaNet: Toward Human Activity Knowledge Engine (CVPR2020) [Code] [Data] [Paper] [YouTube] [bilibili]

• Pairwise (ECCV2018) [Paper]

• Attentional Pooling for Action Recognition (NIPS2017) [Code] [Paper]

• Learning Models for Actions and Person-Object Interactions with Transfer to Question Answering (ECCV2016) [Code] [Paper]

• Contextual Action Recognition with R*CNN (ICCV2015) [Code] [Paper]

• HOCNN (ICCV2015) [Code] [Paper]

• SGAP-Net (AAAI2020) [Paper]

More...

Unseen or zero-shot learning (image-level recognition).

• ICompass (ICCV2021) [Paper], [Code]

• Compositional Learning for Human Object Interaction (ECCV2018) [Paper]

• Zero-Shot Human-Object Interaction Recognition via Affordance Graphs (Sep. 2020) [Paper]

More...

HOI Detection: Instance-based, to detect the human-object pairs and classify the interactions.

• PartMap (ECCV2022), [Paper], [Code]

• K-BAN (arXiv 2022), [Paper]

• SGCN4HOI (arXiv 2022), [Paper]

• HQM (ECCV 2022), [Paper], [Code]

• ODM (arXiv 2022), [Paper]

• SDT (arXiv 2022), [Paper]

• DOQ (CVPR 2022), [Paper], [Code]

• STIP (CVPR 2022), [Paper]

• DT (CVPR 2022), [Paper]

• IF (CVPR 2022), [Paper], [Code]

• CPC (CVPR 2022), [Paper], [Code]

• CATN (arXiv 2022), [Paper]

• SSRT (CVPR 2022), [Paper]

• GEN-VLKT (CVPR 2022), [Paper], [Code]

• MSTR (CVPR 2022), [Paper]

• Iwin (arXiv 2022.3), [Paper]

• RGBM (arXiv 2022.2), [Paper]

• GPV-2 (arXiv 2022.2), [Paper], [Project]

• OC-Immunity (AAAI 2022) [Paper], [Code]

• OCN (AAAI 2022) [Paper], [Code]

• QAHOI (arXiv 2021) [Paper], [Code]

• PhraseHOI (AAAI 2022) [Paper]

• DEFR (arXiv 2021.12) [Paper]

• UPT (arXiv 2021) [Paper], [Code]

• HRNet (TIP 2021) [Paper]

• ACP++ (TIP 2021) [Paper], [Code]

• SG2HOI (ICCV 2021) [Paper]

• CDN (NeurIPS 2021) [Paper], [Code]

• GTNet (arXiv 2021.8) [Paper], [Code]

• HOI-MO-Net (IVC 2021) [Paper]

• IPGN (TIP 2021.7) [Paper]

• SCG (ICCV 2021, SAG, v2) [Paper], [Code]

• Human Object Interaction Detection using Two-Direction Spatial Enhancement and Exclusive Object Prior (arXiv) [Paper]

• PST (ICCV2021) [Paper]

• RR-Net (arXiv 2021.5) [Paper]

• HOTR (CVPR2021) [Paper], [Code]

• GGNet (CVPR2021) [Paper], [Code]

• ATL (CVPR2021) [Paper], [Code]

• FCL (CVPR2021) [Paper], [Code]

• AS-Net (CVPR2021) [Paper], [Code]

• End-to-End Human Object Interaction Detection with HOI Transformer (CVPR2021), [Paper], [Code]

• QPIC (CVPR2021) [Paper], [Code]

• TIN (TPAMI2021) [Paper], [Code]

• IDN (NeurIPS2020) [Paper] [Code]

• DIRV (AAAI2021) [Paper]

• DecAug (AAAI2021) [Paper]

• OSGNet (IEEE Access) [Paper]

• PFNet (CVM) [Paper]

• UniDet (ECCV2020) [Paper]

• DRG (ECCV2020) [Paper] [Code]

• FCMNet (ECCV2020) [Paper]

• Contextual Heterogeneous Graph Network for Human-Object Interaction Detection (ECCV2020) [Paper]

• PD-Net (ECCV2020) [Paper-1] [Paper-2] [Code]

• VCL (ECCV2020) [Paper] [Code]

• ACP (ECCV2020) [Paper] [Code]

• ConsNet (ACMMM2020) [Paper] [Code], HICO-DET Python API: A general Python toolkit for the HICO-DET dataset, including APIs for data loading & processing, human-object pair IoU & NMS calculation, and standard evaluation. [Code] [Documentation]

• Action-Guided Attention Mining and Relation Reasoning Network for Human-Object Interaction Detection (IJCAI2020) [Paper]

• PaStaNet (CVPR2020) [Code] [Data] [Paper] [YouTube] [bilibili]

• DJ-RN (CVPR2020) [Code] [Paper]

• Cascaded Human-Object Interaction Recognition (CVPR2020) [Code] [Paper]

• PPDM (CVPR2020) [Code] [Paper]

• IP-Net (CVPR2020) [Code] [Paper]

• VSGNet (CVPR2020) [Code] [Paper]

• HOID (CVPR2020) [Code] [Paper]

• Diagnosing Rarity in Human-Object Interaction Detection (CVPRW2020) [Paper]

• MLCNet (ICMR2020) [Paper]

• SIGN (ICME2020) [Paper]

• In-GraphNet (IJCAI-PRICAI 2020) [Paper]

• PMFNet(ICCV2019) [Code] [Paper]

• No-Frills (ICCV2019) [Code] [Paper]

• Analogy (ICCV2019) [Code] [Paper]

• RPNN (ICCV2019) [Paper]

• Deep Contextual Attention for Human-Object Interaction Detection (ICCV2019) [Paper]

• Interactiveness (CVPR2019) [Code] [Paper]

• Turbo (AAAI2019) [Paper]

• GPNN (ECCV2018) [Code] [Paper]

• iCAN (BMVC2018) [Code] [Paper]

• InteractNet (CVPR2018) [Paper]

• Scaling Human-Object Interaction Recognition through Zero-Shot Learning (WACV2018) [Paper]

• HO-RCNN (WACV2018) [Code] [Paper]

• VS-GATs (Mar. 2020) [Paper]

• Classifying All Interacting Pairs in a Single Shot (Jan. 2020) [Paper]

• Novel Human-Object Interaction Detection via Adversarial Domain Generalization (May. 2020) [Paper]

• PMN (Jul. 2020) [Paper] [Code]

• SAG (Dec 2020) [Paper] [Code]

• SABRA (Dec 2020) [Paper]

More...

Unseen or zero/low-shot or weakly-supervised learning (instance-level detection).

• THID (CVPR 2022), [Paper], [Code]

• EoID (arXiv 2022), [Paper], [Code]

• SCL (arXiv 2022), [Paper], [Code]

• GEN-VLKT (CVPR 2022), [Paper], [Code]

• OC-Immunity (AAAI 2022), [Paper], [Code]

• CHOID (CVPR 2021), [Paper], [Code]

• Align-Former (BMVC 2021), [Paper]

• Discovering Human Interactions with Large-Vocabulary Objects via Query and Multi-Scale Detection (ICCV2021) [Paper], [Code]

• DGIG-Net (TOC2021) [Paper]

• ATL (CVPR2021) [Paper], [Code]

• FCL (CVPR2021) [Paper], [Code]

• Detecting Human-Object Interaction with Mixed Supervision (WACV 2021) [Paper]

• ConsNet (ACMMM2020) [Paper] [Code]

• Zero-Shot Human-Object Interaction Recognition via Affordance Graphs (Sep. 2020) [Paper]

• VCL (ECCV2020) [Paper] [Code]

• HOID (CVPR2020) [Code] [Paper]

• Novel Human-Object Interaction Detection via Adversarial Domain Generalization (May. 2020) [Paper]

• Analogy (ICCV2019) [Code] [Paper]

• Functional (AAAI2020) [Paper]

• Scaling Human-Object Interaction Recognition through Zero-Shot Learning (WACV2018) [Paper]

More...

Video HOI methods

• SPDTP (arXiv, Jun 2022), [Paper]

• V-HOI (arXiv, Jun 2022), [Paper]

• Detecting Human-Object Relationships in Videos (ICCV2021) [Paper]

• STIGPN (Aug 2021), [Paper], [Code]

• VidHOI (May 2021), [Paper]

• LIGHTEN (ACMMM2020) [Paper] [Code]

• Generating Videos of Zero-Shot Compositions of Actions and Objects (Jul 2020), HOI GAN, [Paper]

• Grounded Human-Object Interaction Hotspots from Video (ICCV2019) [Code] [Paper]

• GPNN (ECCV2018) [Code] [Paper]

More...

Result

PaStaNet-HOI:

Proposed by TIN (TPAMI version, Transferable Interactiveness Network). It is built on HAKE data, includes 110K+ images and 520 HOIs (without the 80 "no_interaction" HOIs of HICO-DET to avoid the incomplete labeling). It has a more severe long-tailed data distribution thus is more difficult.

Detector: COCO pre-trained

HICO-DET:

1) Detector: COCO pre-trained

2) Detector: pre-trained on COCO, fine-tuned on HICO-DET train set (with GT human-object pair boxes) or one-stage detector (point-based, transformer-based)

Finetuned detector would learn to only detect the interactive humans and objects (with interactiveness), thus suppress many wrong pairings (non-interactive human-object pairs) and boost the performance.

3) Ground Truth human-object pair boxes (only evaluating HOI recognition)

4) Interactiveness detection (interactive or not + pair box detection):

5) Enhanced with HAKE:

6) Zero-Shot HOI detection:

Unseen action-object combination scenario (UC)

• * indicates large Visual-Language model pretraining, \eg, CLIP.
• For the details of the setting, please refer to corresponding publications. This is not officially published and might miss some publications. Please find the corresponding publications.

Unseen object scenario (UO)

Unseen action scenario (UA)

Another setting

Ambiguous-HOI

Detector: COCO pre-trained

SWiG-HOI

V-COCO: Scenario1

1) Detector: COCO pre-trained or one-stage detector

2) Enhanced with HAKE:

HOI-COCO:

based on V-COCO

HICO

1) Default

2) Enhanced with HAKE: