Roadmap of MMPose

[hellock]

We keep this issue open to collect feature requests from users and hear your voice. Our monthly release plan is also available here.

You can either:

1. Suggest a new feature by leaving a comment.
2. Vote for a feature request with 👍 or be against with 👎. (Remember that developers are busy and cannot respond to all feature requests, so vote for your most favorable one!)
3. Tell us that you would like to help implement one of the features in the list or review the PRs. (This is the greatest things to hear about!)

[aa12356jm]

i hope that MMPose can support 21 hand landmark detetion, thanks

[jin-s13]

i hope that MMPose can support 21 hand landmark detetion, thanks

Good suggestions! We will add this feature in our TODO list. Thank you.

[jin-s13]

TODO List (continuously updated... [last edit: 2023.1.14]) : Here is a collection of feature requests. Items that have already been implemented in MMPose will be removed from the list.

1. More popular backbones

• [ ] ConvNeXt

2. Add more popular datasets:

• [ ] LSP
• [ ] MPII test set
• [ ] MPII multi-person dataset for bottom-up methods
• [ ] FashionAI
• [ ] DeepFashion2
• [ ] 3dpw
• [ ] MuCo-3DHP

3. More 2d human pose estimation method.

• [ ] CPN
• [ ] OpenPose #651
• [ ] PifPaf
• [ ] ZoomNet
• [ ] OPEC-Net
• [ ] CenterNet
• [ ] MoveNet
• [ ] MIPNet
• [ ] ViTPose #1876

4. More 2d face alignment algorithms.

5. More 3d human pose algorithms.

• [ ] DecoMR
• [ ] PoseFormer #1215
• [ ] C2F #868 #975
• [ ] SmoothNet training codes and models trained on 3DPW and AIST.
• [ ] PoseAug

6. Support 2d video pose estimation and tracking

• [ ] Penn Action
• [ ] LSTM Pose Machine
• [ ] DetTrack
• [ ] KeyTrack
• [ ] DCPose #759
• [ ] Skip-Convolultion #836

7. Support Vehicle pose estimation

• [ ] CarFusion dataset
• [ ] Occlusion-net
• [ ] Apollocar3D

8. Add 3D Pose Consistency Benchmark #828

9. Mano based hand keypoints detection

10. Depth-based 3d hand pose estimation

• [ ] A2J

11. Multi-view 3d pose estimation

• [ ] Graph-Based 3D Multi-Person Pose Estimation Using Multi-View Images
• [ ] 3DMPPE PoseNet #964
• [ ] Learnable Triangulation of Human Pose #864
• [ ] Epipolar Transformers #864
• [ ] Cross view fusion for 3d human pose estimation #864

12. Support memonger

13. Support Pytorch AMP training #339

14. Hyperparameter tuner Optuna

15. Support Unity plugin

16. print loss during evaluation. #333

17. Quantization Aware Training #359

18. Easier Usage (API)

19. Export to Torchscript #576

[gachiemchiep]

Would you mind add https://github.com/open-mmlab/mmpose/issues/31#issuecomment-663334223 to the TODO list.

[jin-s13]

Would you mind add #31 (comment) to the TODO list.

Sure.

[innerlee]

Speed up inference https://github.com/open-mmlab/mmpose/issues/40

[jin-s13]

Support video pose estimation #67

[pablovela5620]

Would be great to add support for whole body pose estimation dataset (body+face+hands) via COCO-WholeBody

[lisa676]

Also add support for MPII in mmdetection.

[OasisYang]

It would be great to add support for pose tracking dataset i.e. posetrack2017/2018.

[flynnamy]

Support to convert pytorch model to onnx by the way.Thx！

[innerlee]

@OasisYang could you elaborate? is loading the data and processing on a frame basis is enough, or you want the tracking part also?

@flynnamy sounds like a request for a general tool. maybe we can provide such tools for the whole mm-series (just saying, not a confirmation).

[OasisYang]

@innerlee If possible, adding both data loading and tracking part would be great. However, the tracking part seems a little bit complicate and always comes with some extra modules. Maybe, the first step is basically to support the data loading and processing. Thanks

@OasisYang could you elaborate? is loading the data and processing on a frame basis is enough, or you want the tracking part also?

@flynnamy sounds like a request for a general tool. maybe we can provide such tools for the whole mm-series (just saying, not a confirmation).

[jin-s13]

Support ShuffleNet V2 & MobileNet V3 backbones. #94

[pablovela5620]

Support for InterHand2.6

[Witek-]

Add Yolov4 and OpenPose

[fhir-obesity]

Please make it possible to obtain estimated heatmaps from methods

[innerlee]

@hamedcan could you explain more about the usage? do you want a visualizer of heatmaps during training, or a visualization tool for demo, or anything else?

[san-guy]

Bottom up for MPII dataset?

[fhir-obesity]

@innerlee First, I really want to thank you for the MMPose. It really helped me. I want to compare different models' performance on hard poses. So I need to be able to observe generated heatmaps. I want a visualization tool for demo.

[HuangJunJie2017]

support memonger : https://github.com/Lyken17/pytorch-memonger

[jin-s13]

Support multi-head networks #219

[flynnamy]

Please support mpii_trb demo and mpi_inf_3dhp datasets!

[yanglilian]

Support 3d hand keypoint estimation!!!!!!

[yulong314]

Support log info when dataset is tinty, #333

[YibinXie]

Support PyTorch AMP training, thanks. #339

[YibinXie]

Support GCN-based methods for refining top-down results. https://arxiv.org/pdf/2003.10506v3.pdf https://github.com/lingtengqiu/OPEC-Net https://arxiv.org/abs/2007.10599

[pablovela5620]

Would be great to see the integration of a hyperparameter tuner like Optuna

[MaxGodTier]

A Unity plugin would be amazing to have, using json input data and/or real-time pose estimation with a webcam and seeing it reflected on a 3D model.

[innerlee]

@MaxGodTier do you have experience in developing unity plugin? contributions are welcome :D

[MaxGodTier]

I don't, but a dirty implementation may be possible using an existing repo , it reads pose data from simple text files each representing a single frame , I see two solutions: (1) If pose_results from mmpose were translated into the same format expected from that repo, it will work out of the box without needing to change a single line of code or (2) edit the repo code (C#) to use mmpose rules instead of theirs.

[aa12356jm]

Quantization Aware Training for models to get the int8 models ,int8 models will greatly improve inference speed #359,thanks

[zgs731]

Support DetTrack and KeyTrack. http://arxiv.org/abs/2003.13743 & https://arxiv.org/abs/1912.02323

[pablovela5620]

Albumentations augmetnations similar to mmclassification

[aa12356jm]

i hope that MMPose can support 3D hand landmarks detetion, thanks

[rhiver]

Does MMPose support Single Person Pose Estimation? Currently I found only multi-person versions are supported.

[innerlee]

@rhiver single is a case of multi

[rhiver]

@rhiver single is a case of multi

Sort of. But multi-person version has two stages, person detection and pose estimation, which have to infer on two models. So this method doesn't work for realtime pose estimation in mobile devices since it takes too long on the inference. MobileNetV2 is good enough for simple pose estimation. But for best FPS, it's better to let it do both single person detection and pose estimation.

[flynnamy]

I see it has supported heatmap method of face datasets now. Please support regression method of face dataset!

[jin-s13]

I see it has supported heatmap method of face datasets now. Please support regression method of face dataset!

Do you have any recommended papers/codes ?

[flynnamy]

I see it has supported heatmap method of face datasets now. Please support regression method of face dataset!

Do you have any recommended papers/codes ?

Yes,wingloss:https://arxiv.org/pdf/1711.06753.pdf, and GCN+softwing loss: https://arxiv.org/pdf/2006.11697.pdf

[innerlee]

Support FashionAI https://tianchi.aliyun.com/competition/entrance/231648/introduction

[CHRIBUR0309]

I want to use mmpose with the pypi package much more easily than now; such as:

from mmpose import top_down

top_down("darkpose", "COCO_wholebody", video_path="hoge.mp4", output_json_title="hoge") # Analyze hoge.mp4 with COCO wholebody on darkpose and output the result as hoge/hoge000000000000.json, hoge/hoge000000000001.json, hoge/hoge000000000002.json, ....

[yeshaokai]

MPII multi-person dataset for bottom-up methods is really needed!

[aa12356jm]

hi ,can you add handlandmark filtering algorithm for eliminating handlandmark jittering in videos? thanks

[dineshreddy91]

Adding Vehicle pose estimation to the pipe line using CarFusion dataset. Similar to Occlusion-net, and Apollocar3D.

[simba0703]

Export to Torchscript !

[pablovela5620]

Lite-HRNet, its already built with mmpose, so including into the main repo should be super simple. Would be amazing if it could work with the pytorch2onnx tool for deployment

[ioctl-user]

Please support Halpe data set: https://github.com/Fang-Haoshu/Halpe-FullBody

It has 3 useful points in addition to the COCO-WholeBody.

[rubeea]

Hi everyone,

I intend to create my own keypoints dataset with 3 points of interest (two endpoints and one center point). Can anyone kindly help me on how I can create annotations to be loaded into mmpose? Because I believe that the repo is based on mmcv, how can I get my own dataloader? Any help in this regard will be highly appreciated. Thank you

[jin-s13]

Support 3dpw dataset #682

[aqsc]

Do you have any plans for the mano based hand keypoints detection? Also optimization with the IK loss

[AmitMY]

Add 3D Pose Consistency Benchmark - https://github.com/open-mmlab/mmpose/issues/828

[znoland3]

Add https://github.com/mks0601/3DMPPE_POSENET_RELEASE into MMPose

[Broken1334]

It would be nice to add "PoseFormer". It based on VideoPose3D, which already supported.

[ee-lynn]

It would be nice to add "CenterNet". it is a bottom up based 2d human pose estimation method and it groups keypoints of one person by combine regression and heatmap of keypoints which is quite different from associated embedding and affinity fields

[APeiZou]

Blog:Next-Generation Pose Detection with MoveNet and TensorFlow.js 这里有movenet的简单介绍，https://storage.googleapis.com/movenet/MoveNet.SinglePose%20Model%20Card.pdf

[canghaiyunfan]

Add RLE into MMPose

[francescotaioli]

Background : 3d pose estimation (with video generation) with a high number of people (es: official video, minute 00:19 sec, but with a lot of people

Result video: the original video is put on the top-left, with the subsuquent 3d pose of the people on the right. If there are a lot of people, the final video has strange resolution (i.e 6000x400) because every people detected is on put on the same row.

What could be improve: split the people 3d pose visualization into multiple row

[piercus]

It would be great to have a 'score_per_joint' option in test_cfg in order to output one score per joint, instead of having only a global score for the pose, my use case is related to associative embedding

[liwenssss]

update Interhand2.6M dataset which contains MANO hand mesh parameters.......

[walsvid]

It would be nice to have Depth-Based 3D Hand Pose Estimation methods like A2J.

[vra]

It would be great to have SmoothNet trained on 3DPW and AIST++ :)

[qinb]

It would be nice to add SmoothNet training code about pose estimation, hoping it could easily retrain on my own dataset.

[Joyako]

3D Human Mesh frankmocap

[ly015]

3D Human Mesh frankmocap

Thanks for your feedback. 3D human mesh recovery is no longer supported in MMPose. We have MMHuman3D for this task and you are welcome to submit an issue there about your request.

[yshMars]

It would be so helpful for better analysis if AP for each type of body joints are printed, for example 17 AP value for 17 kinds of body joints are given when inferencing a model in MS COCO body-keypoint dataset.

[levan92]

Will be really helpful to implement MIPNet into mmpose:

• It is particularly useful to tackle data where there are crowded/highly occluded humans. Was previously the SOTA on OCHuman before ViTPose came along. Within the realms of convnets, it should still be the SOTA, and it seems like the idea is general enough to be applied to different types of backbones.

[levan92]

Also similar to #1389 request, will be nice to integrate ViTPose into mmpose. ViTPose is already implemented in mmpose, so I expect integration to be much easier 😄

[jin-s13]

It would be so helpful for better analysis if AP for each type of body joints are printed, for example 17 AP value for 17 kinds of body joints are given when inferencing a model in MS COCO body-keypoint dataset.

@yshMars Already supported in https://github.com/open-mmlab/mmpose/pull/1170

[liuzhuang13]

It would be nice to support ConvNeXt backbones. It is a very simple model that is purely convolutional. They can serve as a drop-in replacement for ResNet or Swin Transformer architectures. ImageNet-22k pretrained ConvNeXt variants are considered state-of-the-art in this regime.

Official code: https://github.com/facebookresearch/ConvNeXt ConvNeXt was also implemented in the mmsegmentation and mmdetection libraries.

Thanks!