Making sense of SMPLOutput results

[UnrealXinda]

Hi, could someone help me with interpreting the results of SMPLOutput? I can get a rough estimate of what each field means. But I can't seme to find documentation on what coordinate space those values are under. Specifically, are they in world space? Component space (wrt to the character's root or whatever fixed point it is)? Local space (with relative to its parent bone/joint's coordinate system)? The fields that I'm trying to understand are:

• body_pose (3x3 rotation matrices). I'm assuming they're the bones'/joints' rotation. But in which space?
• global_orient (3x3 rotation matrices). Same as above.
• joints (vectors of positions). This seems like joint positions. I'm not sure which space they're in. Also, with 23 joints passed into the model, it generates 45 joint positions. I'm not fully understand where the discrepancy comes from.

Thank you!

[MingCongSu]

Same question

I run the examples/demo.py to display SMPL model, but why the Joints shape is (45, 3) but (24, 3)? I am curious about the output

[AmoghTiwari]

I also have this doubt. It would be great if someone could clarify this. Further, is there any joint regressor available which can give me these 45 joints from the SMPL vertices? (I am evaluating an existing method which directly regresses vertex positions. However, to evaluate it with the same script as other methods, I would need to get these 45 joints using a regressor).

[suminmin]

Hi, I believe this is related to the information discussed below.

As far as I examined, it appears that the number of joints can be modified using the joint_mapper, which can be configured within each SMPL class. https://github.com/vchoutas/smplx/blob/1265df7ba545e8b00f72e7c557c766e15c71632f/smplx/body_models.py#L489

For example, in SMPLify-X, they've implemented joint_mapper and VertexJointSelector to convert SMPL-X joints and vertices to the OpenPose25 detected keypoints. https://github.com/vchoutas/smplify-x/blob/3e11ff1daed20c88cd00239abf5b9fc7ba856bb6/smplifyx/utils.py#L98

The correspondences between SMPL-X vertex indices and OpenPose keypoint names are defined here. https://github.com/vchoutas/smplx/blob/1265df7ba545e8b00f72e7c557c766e15c71632f/smplx/vertex_ids.py

Basically SMPL-X outputs (body joints + hand joints + face joints, 3), on the other hand SMPLX class with this joint_mapper outputs (25, 3) joints for OpenPose definition. Therefore, the joint_mapper allows for the adjustment of the number of outputted joints to any desired quantity, whether more or less.

I hope this information would be helpful.

[suminmin]

Update I figured out why it happens. As I thought, the thing comes from VertexJointSelector.

45 joints = (SMPL joints + face vertices + feet vertices + both hand vertices) = (24 + 5 + 6 + 5*2) https://github.com/vchoutas/smplx/blob/1265df7ba545e8b00f72e7c557c766e15c71632f/smplx/vertex_joint_selector.py#L29

The default argments of VertexJointSelector are set to both use_hands= True, use_hands = True. On top of that, wheter both are set True or False, at least face_keyp_idxs is added, which means outputed joints should be (24 + 5, 3) joints.

I think just extracting first (24, 3) tensors would be OK in practice.

class VertexJointSelector(nn.Module):

    def __init__(self, vertex_ids=None,
                 use_hands=True,
                 use_feet_keypoints=True, **kwargs):
        super(VertexJointSelector, self).__init__()

        extra_joints_idxs = []

        face_keyp_idxs = np.array([
            vertex_ids['nose'],
            vertex_ids['reye'],
            vertex_ids['leye'],
            vertex_ids['rear'],
            vertex_ids['lear']], dtype=np.int64)

        extra_joints_idxs = np.concatenate([extra_joints_idxs,
                                            face_keyp_idxs])

        if use_feet_keypoints:
            feet_keyp_idxs = np.array([vertex_ids['LBigToe'],
                                       vertex_ids['LSmallToe'],
                                       vertex_ids['LHeel'],
                                       vertex_ids['RBigToe'],
                                       vertex_ids['RSmallToe'],
                                       vertex_ids['RHeel']], dtype=np.int32)

            extra_joints_idxs = np.concatenate(
                [extra_joints_idxs, feet_keyp_idxs])

        if use_hands:
            self.tip_names = ['thumb', 'index', 'middle', 'ring', 'pinky']

            tips_idxs = []
            for hand_id in ['l', 'r']:
                for tip_name in self.tip_names:
                    tips_idxs.append(vertex_ids[hand_id + tip_name])

            extra_joints_idxs = np.concatenate(
                [extra_joints_idxs, tips_idxs])

        self.register_buffer('extra_joints_idxs',
                             to_tensor(extra_joints_idxs, dtype=torch.long))