Is it possible to train rokoko dataset in mmpose?

[fredyu]

trafficstars

Thanks for your feature request and we will review and plan for it when necessary. If you feel we have helped you, give us a STAR! :satisfied:

Steps

1. Check if the feature has been requested in the meta issue, and if so, click thumb up button.
2. Post the feature request in the meta issue, if it is new.

Describe the feature

Motivation

A clear and concise description of the motivation of the feature.

1. Ex1. It is inconvenient when [....].
2. Ex2. There is a recent paper [....], which is very helpful for [....].

Related resources

If there is an official code released or third-party implementations, please also provide the information here, which would be very helpful.

Additional context

Add any other context or screenshots about the feature request here. If you would like to implement the feature and create a PR, please leave a comment here and that would be much appreciated.

[jin-s13]

Could you please provide more details about rokoko dataset?

[fredyu]

Thanks Jin-s13 for the quick response. The ROKOKO dataset is saved in a csv file, format as following:

Timestamp	Root_P_x	Root_P_y	Root_P_z	Root_Q_x	Root_Q_y	Root_Q_z	Root_Q_w	Hips_P_x	Hips_P_y	Hips_P_z	Hips_Q_x	Hips_Q_y	Hips_Q_z. .............
0	0	0	0	6.657903e-08	0	0	1	-0.007210765	0.909949	-0.002047792	0.0149896	0.007403604	0.001145249
0	0	0	0	6.657903e-08	0	0	1	-0.007210765	0.909949	-0.002047792	0.0149896	0.007403604	0.001145249
0	0	0	0	6.657903e-08	0	0	1	-0.007210765	0.909949	-0.002047792	0.0149896	0.007403604	0.001145249
10.00977	0	0	0	6.657903e-08	0	0	1	-0.007251054	0.9095271	-0.002021573	0.01500696	0.007402699	0.001134259
20.01953	0	0	0	6.657903e-08	0	0	1	-0.007284701	0.909778	-0.001934813	0.01497277	0.007376918	0.001162205
30.0293	0	0	0	6.657903e-08	0	0	1	-0.007382753	0.9097246	-0.001832062	0.0150325	0.007342283	0.001128938
39.79492	0	0	0	6.657903e-08	0	0	1	-0.007468735	0.9097522	-0.001758403	0.01517074	0.007407371	0.001001961
49.80469	0	0	0	6.657903e-08	0	0	1	-0.007545079	0.9097492	-0.001727905	0.01532079	0.007468593	0.0009430593

Each line includes 183 values: Timestamp | Root_P_x | Root_P_y | Root_P_z | Root_Q_x | Root_Q_y | Root_Q_z | Root_Q_w | Hips_P_x | Hips_P_y | Hips_P_z | Hips_Q_x | Hips_Q_y | Hips_Q_z | Hips_Q_w | LeftThigh_P_x | LeftThigh_P_y | LeftThigh_P_z | LeftThigh_Q_x | LeftThigh_Q_y | LeftThigh_Q_z | LeftThigh_Q_w | LeftShin_P_x | LeftShin_P_y | LeftShin_P_z | LeftShin_Q_x | LeftShin_Q_y | LeftShin_Q_z | LeftShin_Q_w | LeftFoot_P_x | LeftFoot_P_y | LeftFoot_P_z | LeftFoot_Q_x | LeftFoot_Q_y | LeftFoot_Q_z | LeftFoot_Q_w | LeftToe_P_x | LeftToe_P_y | LeftToe_P_z | LeftToe_Q_x | LeftToe_Q_y | LeftToe_Q_z | LeftToe_Q_w | LeftToeTip_P_x | LeftToeTip_P_y | LeftToeTip_P_z | LeftToeTip_Q_x | LeftToeTip_Q_y | LeftToeTip_Q_z | LeftToeTip_Q_w | RightThigh_P_x | RightThigh_P_y | RightThigh_P_z | RightThigh_Q_x | RightThigh_Q_y | RightThigh_Q_z | RightThigh_Q_w | RightShin_P_x | RightShin_P_y | RightShin_P_z | RightShin_Q_x | RightShin_Q_y | RightShin_Q_z | RightShin_Q_w | RightFoot_P_x | RightFoot_P_y | RightFoot_P_z | RightFoot_Q_x | RightFoot_Q_y | RightFoot_Q_z | RightFoot_Q_w | RightToe_P_x | RightToe_P_y | RightToe_P_z | RightToe_Q_x | RightToe_Q_y | RightToe_Q_z | RightToe_Q_w | RightToeTip_P_x | RightToeTip_P_y | RightToeTip_P_z | RightToeTip_Q_x | RightToeTip_Q_y | RightToeTip_Q_z | RightToeTip_Q_w | Spine1_P_x | Spine1_P_y | Spine1_P_z | Spine1_Q_x | Spine1_Q_y | Spine1_Q_z | Spine1_Q_w | Spine2_P_x | Spine2_P_y | Spine2_P_z | Spine2_Q_x | Spine2_Q_y | Spine2_Q_z | Spine2_Q_w | Spine3_P_x | Spine3_P_y | Spine3_P_z | Spine3_Q_x | Spine3_Q_y | Spine3_Q_z | Spine3_Q_w | Spine4_P_x | Spine4_P_y | Spine4_P_z | Spine4_Q_x | Spine4_Q_y | Spine4_Q_z | Spine4_Q_w | LeftShoulder_P_x | LeftShoulder_P_y | LeftShoulder_P_z | LeftShoulder_Q_x | LeftShoulder_Q_y | LeftShoulder_Q_z | LeftShoulder_Q_w | LeftArm_P_x | LeftArm_P_y | LeftArm_P_z | LeftArm_Q_x | LeftArm_Q_y | LeftArm_Q_z | LeftArm_Q_w | LeftForeArm_P_x | LeftForeArm_P_y | LeftForeArm_P_z | LeftForeArm_Q_x | LeftForeArm_Q_y | LeftForeArm_Q_z | LeftForeArm_Q_w | LeftHand_P_x | LeftHand_P_y | LeftHand_P_z | LeftHand_Q_x | LeftHand_Q_y | LeftHand_Q_z | LeftHand_Q_w | Neck_P_x | Neck_P_y | Neck_P_z | Neck_Q_x | Neck_Q_y | Neck_Q_z | Neck_Q_w | Head_P_x | Head_P_y | Head_P_z | Head_Q_x | Head_Q_y | Head_Q_z | Head_Q_w | RightShoulder_P_x | RightShoulder_P_y | RightShoulder_P_z | RightShoulder_Q_x | RightShoulder_Q_y | RightShoulder_Q_z | RightShoulder_Q_w | RightArm_P_x | RightArm_P_y | RightArm_P_z | RightArm_Q_x | RightArm_Q_y | RightArm_Q_z | RightArm_Q_w | RightForeArm_P_x | RightForeArm_P_y | RightForeArm_P_z | RightForeArm_Q_x | RightForeArm_Q_y | RightForeArm_Q_z | RightForeArm_Q_w | RightHand_P_x | RightHand_P_y | RightHand_P_z | RightHand_Q_x | RightHand_Q_y | RightHand_Q_z | RightHand_Q_w

Based on this ROKOKO file, we have rendered out PNG file from Maya for each line (timestamp):

[fredyu]

and we can visualize it like:

The 3d to 2d projection is not exactly matched since we don't the camera parameters when rendering on Maya.

[ly015]

Thanks for using MMPose. Please refer to our tutorial about how to add a new dataset in MMpose. This dataset seems to include 2D/3D keypoints, rendered synthetic images, and camera parameters. You can refer to the implementations of COCO dataset the Human3.6M dataset for 2D keypoint detection and 3D keypoint detection respectively.