axis in SMPL result

[ahsan3803]

I'm trying to animate 3D character in Unity using SMPL 24 joints data. Everything is going well but the only problem is that there is different in Unity and SMPL final data axis.

Below axis are from SMPL data (I draw 3D skeleton using 24 joints from CenterHMR)

Below image is about axis in Unity

Seems that we need to change the Y and Z axis in SMPL results.

So suggest me that we should swap/change axis (Y and Z) in j3d_smpl24 list that I got from CenterHMR from here or in Unity side?

[Arthur151]

Good question. The j3d_smpl24 and kp3d_smpl24_results are both in shape B x 24 x 3. The last dimension is in order of (x,y,z). While the results[img_path][subject_idx]['j3d_smpl24'] is in shape 24 x 3. If you want to swap axis (Y and Z), simply add a line below the line you quoted. results[img_path][subject_idx]['j3d_smpl24'] = results[img_path][subject_idx]['j3d_smpl24'][:,[0,2,1]]

[ahsan3803]

Thanks for you reply. I have another question. As I mentioned before that I'm trying to animate Avatar in Unity using joints position from CenterHMR. From CenterHMR we can get SMPL 24 joints or OpenPose 25 joints, that are already discussed in #21, but in both cases there is no top_head joint which is causing problem to head movement. Check the attached screenshot that I tested with CenterHMR (no top_head) and with other model (with top_head joint from SPIN).

I read about SPIN joints from which we can get Head joint (from h36m) and top_head joint (LSP). Many models are using SPIN regressor and joints for better motion but from CenterHMR I'm unable to get better motion in case of head even CenterHMR can output head joint in case of 24 SMPL joints and nose joints in case of OpenPose joints but still it can't solve head movement as compared to SPIN joints.

How we can get SPIN joints in CenterHMR by using SPIN regressor? It will need training with CenterHMR?

I think this functionality should be added in CenterHMR as it already can output OpenPose keypoints.

[Arthur151]

Sorry for the delayed reply! At this moment, I have something urgent at hand. But I will look into the SPIN joints you mentioned later. I think it wouldn't need any training.

[ahsan3803]

It would be appreciated

[lisa676]

I am also waiting for this next commit.

[Arthur151]

I have submitted a commit to support the extra 24 joints in SPIN (compared with 25 openpose joints we already support). With this commit, the result files will contain a key named "j3d_spin24", which is what you are asking for.

Let me know if there are any problems. Besides, have you converted the output file into bvh? Would you want to share the detailed process with other developers? It seems important for the others. While I am not familiar with these parts. Best.

[lisa676]

Thanks for this great commit. I am also looking for BVH step. @ahsan3803 would you like to update about it?

[ahsan3803]

@Arthur151 Thanks for updating the commit for SPIN joints.

Sorry for late reply. I didn't complete the BVH task, still I'm working on it. I will update ASAP whenever I will complete it.

Regarding SPIN joints, I tested them and seems that joints are wrong. SPIN joints order as you mentioned here in constants file is right if we ignore OpenPose joints. I'm getting the wrong joints, to verify it, I draw the skeleton for one frame after getting SPIN joints from here.

Below is the skeletal result for SPIN joints from CenterHMR.

From above attached result, we can see that some joints Neck_LSP[12], Head_top[13], Jaw_H36M[17], Head_H36M[18] are missing and some joints Thorax_MPII[15], Spine_H36M[16] are not at right place. Furthermore the other problem I noticed that, right_knee, left_knee, right_ankle and left_ankle are at two positions. right_knee is at [1] and [15]. left_knee is at [4] and [14]. right_ankle is at [0] and [18]. left_ankle is at [5] and [17].

Below is the complete order of currently joints for SPIN in CenterHMR and it seems that this order is not right according to SPIN 24 joints [0] right_ankle [1] right_knee [2] pelvis [3] left_hip [4] left_knee [5] left_ankle [6] right_wrist [7] right_elbow [8] right_shoulder [9] left_shoulder [10] left_elbow [11] left_wrist [12] right_hip [13] spine [14] left_knee [15] right_knee [16] thorax [17] left_ankle [18] right_ankle [19] nose [20] left_eye [21] right_eye [22] left_ear [23] right_ear

Following attached screenshot is the skeleton limbs and joints connection for above attached skeletal result.

I have tried my best to explain all the problems. Let me know if anything is creating confusion, I will explain in more detail with visual results.

[Arthur151]

Sorry for the bug. I have uploaded a new commit to fix it.

Now the joint order is just as the same as what I define in constrants.py.

[ahsan3803]

Thanks, it's working now. It was minor bug.

[Arthur151]

Thanks for your detailed bug report. I have tested the code at a different codebase. There were something wrong while I pasted the changes to this repository.

At my service. Aha~

[ahsan3803]

It happens to a good developer 😜

[kenchikuliu]

Thanks for your detailed bug report. I have tested the code at a different codebase. There were something wrong while I pasted the changes to this repository.

At my service. Aha~

thx for ur job，my smpl model is just a mesh(obj file, only mesh and texture without skeleton and weights), how I can get a rigged smpl model(fbx) from my own smpl model(obj file) to replace basicModel_m_lbs_10_207_0_v1.0.2.fbx?

[Arthur151]

SMPL is a statistical linear skinned model. Every smpl mesh is just a state of the model in a specific parameters. Please refer to https://smpl.is.tue.mpg.de/ for the details. I am afraid you may have to regress the pose and shape parameters from your smpl mesh via iterative fitting the model. Then you can drive the fbx model to the state of your mesh.

[SajjadAemmi]

Good question. The j3d_smpl24 and kp3d_smpl24_results are both in shape B x 24 x 3. The last dimension is in order of (x,y,z). While the results[img_path][subject_idx]['j3d_smpl24'] is in shape 24 x 3. If you want to swap axis (Y and Z), simply add a line below the line you quoted. results[img_path][subject_idx]['j3d_smpl24'] = results[img_path][subject_idx]['j3d_smpl24'][:,[0,2,1]]

this line and this file are not exist in new version. what is the new place?

[Arthur151]

Hi, there. If you want to switch the dimention of estimated 3D keypoints. You can add a line before this line: https://github.com/Arthur151/ROMP/blob/42f40fdc7be6a282657dccff7623c5872ec947ee/simple_romp/romp/main.py#L169 like:

outputs['joints'] = outputs['joints'][:, [0,2,1]]

The tracking_ids of all subjects in the image are outputs['track_ids']. You can use it to get the motion sequence of each subjects.

[SajjadAemmi]

@Arthur151 When you try to execute the following command, the function you sent will not be executed

python3 -u -m romp.predict.image --configs_yml='configs/image.yml'

[imsatomi]

@Arthur151 Thanks for updating the commit for SPIN joints.

Sorry for late reply. I didn't complete the BVH task, still I'm working on it. I will update ASAP whenever I will complete it.

Regarding SPIN joints, I tested them and seems that joints are wrong. SPIN joints order as you mentioned here in constants file is right if we ignore OpenPose joints. I'm getting the wrong joints, to verify it, I draw the skeleton for one frame after getting SPIN joints from here.

Below is the skeletal result for SPIN joints from CenterHMR.

From above attached result, we can see that some joints Neck_LSP[12], Head_top[13], Jaw_H36M[17], Head_H36M[18] are missing and some joints Thorax_MPII[15], Spine_H36M[16] are not at right place. Furthermore the other problem I noticed that, right_knee, left_knee, right_ankle and left_ankle are at two positions. right_knee is at [1] and [15]. left_knee is at [4] and [14]. right_ankle is at [0] and [18]. left_ankle is at [5] and [17].

Below is the complete order of currently joints for SPIN in CenterHMR and it seems that this order is not right according to SPIN 24 joints [0] right_ankle [1] right_knee [2] pelvis [3] left_hip [4] left_knee [5] left_ankle [6] right_wrist [7] right_elbow [8] right_shoulder [9] left_shoulder [10] left_elbow [11] left_wrist [12] right_hip [13] spine [14] left_knee [15] right_knee [16] thorax [17] left_ankle [18] right_ankle [19] nose [20] left_eye [21] right_eye [22] left_ear [23] right_ear

Following attached screenshot is the skeleton limbs and joints connection for above attached skeletal result.

I have tried my best to explain all the problems. Let me know if anything is creating confusion, I will explain in more detail with visual results.

@ahsan3803 @Arthur151 What a great job! I want to plot the 3D skeleton of my video, can I get the bvh file of the object? Could you tell me how can I get the 3d points file to plot the skeleton of my video?

I'm a bit ashamed to ask this but I'm so glad if you teach me something!