Feature Request: SMPL format export

[jaykup1]

Exported .npy has SMPL parameters but it would be great if it also get exported as pkl file with SMPL parameters.

[GuyTevet]

Hi @jaykup1 ,

You can replace https://github.com/GuyTevet/motion-diffusion-model/blob/14d0f7976e15ed232775688a19ff38f4b2926c08/visualize/vis_utils.py#L66

With pickle.dump(*) if that was your intention.

[jjp9624022]

this is blender addon to opt smpl model here https://github.com/Meshcapade/SMPL_blender_addon The last version allows import animations in .npz format how to trans？It would be great if it could work

[jaykup1]

Hi @jaykup1 ,

You can replace

https://github.com/GuyTevet/motion-diffusion-model/blob/14d0f7976e15ed232775688a19ff38f4b2926c08/visualize/vis_utils.py#L66

With pickle.dump(*) if that was your intention.

Not exactly: https://meshcapade.wiki/SMPL

Anyway to export with this skeleton layout? (so we can use https://github.com/softcat477/SMPL-to-FBX to convert that to FBX for game engines)

[TREE-Ind]

Definitely going to try out the SMPL to FBX converter later tonight. As a workaround we've had great success simply running the rendered mesh sequence through AI mocap solutions to get game ready rigged animations, Plask, Deepmotion, etc.

[jjp9624022]

Hi @jaykup1 , You can replace https://github.com/GuyTevet/motion-diffusion-model/blob/14d0f7976e15ed232775688a19ff38f4b2926c08/visualize/vis_utils.py#L66

With pickle.dump(*) if that was your intention.

Not exactly: https://meshcapade.wiki/SMPL

Anyway to export with this skeleton layout? (so we can use https://github.com/softcat477/SMPL-to-FBX to convert that to FBX for game engines)

not work! different shape results.npy:motion (3, 22, 3, 120) SMPL-to-FBX sample.pkl motion (127, 72)

[jaykup1]

not work! different shape results.npy:motion (3, 22, 3, 120) SMPL-to-FBX sample.pkl motion (127, 72)

The export file has to be rearranged to SMPL skeleton layout before fbx conversion, help would be greatly appreciated.

[GuyTevet]

I'll try to clarify, please let me know if it helps:

results.npy - only includes joint positions - this is not what you are looking for.

after running visualize.render_mesh you will get sample##_rep##_smpl_params.npy with smpl parameters. In details:

            'motion': [25, 6, frames], - this is both SMPL, thetas, and root translation, you can ignore it.
            'thetas': [24, 6, frames] - SMPL thetas represented in 6d rotations
            'root_translation': [3, frames] - Root translation
            'faces':  - SMPL faces list
            'vertices': - SMPL vertices locations per frame
            'text': - text prompt
            'length': - number of frames

• I think 72 stands for 24 joint * 3 (axis angle representation)
• The fields relevant for you are thetas and root_translation you can ignore the rest.
• Alternatively, instead of keyframing SMLP joints you can directly keyframe the mesh's vertices locations, they are also provided in this .npy file

Can you detail here the data structure that the SMPL addon is expecting, and we will try to code an adapter between the two formats (and of course welcome you to send a pull request if you figure it out yourself).

Hope this helps,

[jaykup1]

Thank you for the reply,

• Yes 72 stands for 24 joint * 3 dimension for the rotation vector.
• The pkl file contains a dictionary with: smpl_poses and smpl_trans:
• 'smpl_trans': [frames, 3] - Root translation
• 'smpl_poses' : [frames, 72] - thetas
  • Joint order expected is: 0:Pelves 1:L_hip 2:R_hip 3:Spine1 4:L_Knee 5:R_Knee 6:Spine2 7:L_Ankle 8:R_Ankle 9:Spine3 10:L_Foot 11:R_Foot 12:Neck 13:L_Collar 14:R_Collar 15:Head 16:L_Shoulder 17:R_Shoulder 18:L_Elbow 19:R_Elbow 20:L_Wrist 21:R_Wrist 22:L_Hand 23:R_Hand

[GuyTevet]

Cool! If that's the case, the thetas and root_translation have all the information you need. You will just need to convert 6d into axis angles. One way to do so is to concatenate those two functions in our repo:

https://github.com/GuyTevet/motion-diffusion-model/blob/14d0f7976e15ed232775688a19ff38f4b2926c08/utils/rotation_conversions.py#L513-L534

https://github.com/GuyTevet/motion-diffusion-model/blob/14d0f7976e15ed232775688a19ff38f4b2926c08/utils/rotation_conversions.py#L434-L447

[Tony2371]

So, will there be a script that converts sample##_rep##_smpl_params.npy to FBX or something like that?

BTW, awesome project, guys! Thanks a lot for your efforts.

[amb]

results.npy - only includes joint positions - this is not what you are looking for.

after running visualize.render_mesh you will get sample##_rep##_smpl_params.npy with smpl parameters. In details:

Does this process add new information? I've been building armatures and animations just from the results.npy manually with some quaternion magic. Would sample##_rep##_smpl_params.npy yield better results?

[TREE-Ind]

@amb very cool! Are you able to share any insight or code snippet as to how you accomplished this? We're getting close but it would help us greatly, no worries either way thx.

[amb]

@TREE-Ind Sure, it's just a proof of concept Blender script tho. Here you go: https://gist.github.com/amb/69cc8396bc61cc59a7e819aed6b21f34 The script has "import bpy" so technically it's also GPL3 license.

[TREE-Ind]

@amb Wow nice work :D

[jjp9624022]

@TREE-Ind Sure, it's just a proof of concept Blender script tho. Here you go: https://gist.github.com/amb/69cc8396bc61cc59a7e819aed6b21f34 The script has "import bpy" so technically it's also GPL3 license.

nice work! but....... Like you said ，# TODO: Is this really the best way to create armature in Blender? you can trans the pose from the initial posture of the human skeleton someting like rigfly

[jaykup1]

Work in progess...:

@GuyTevet How can I get the 6d out of 'thetas': self.motions['motion'][0, :-1, :, :self.real_num_frames], for rotation_6d_to_matrix ?

[GuyTevet]

@jaykup1 those are the 6d rotations, but in SMPL convention. I can make a guess why this is not working - (1) it can be an issue with the conversion (2) SMPL angles are relative to SMPL rest pose (T-pose I think), maybe the rest pose of your character is different. In this case, I can suggest, you put the SMPL character (using SMPL addon) side by side with your character, and align between them.

You said rotation_6d_to_matrix doesn't work well, what exactly is the issue there?

[GuyTevet]

@jaykup1 There is knowledge regarding how to convert SMPL angles to other characters (not in my brain, unfortunately). I suggest contacting the SMPL team or watching their tutorials on youtube. If you manage to resolve this, I will be very happy to know about it!

[jaykup1]

@jaykup1 those are the 6d rotations, but in SMPL convention. I can make a guess why this is not working - (1) it can be an issue with the conversion (2) SMPL angles are relative to SMPL rest pose (T-pose I think), maybe the rest pose of your character is different. In this case, I can suggest, you put the SMPL character (using SMPL addon) side by side with your character, and align between them.

You said rotation_6d_to_matrix doesn't work well, what exactly is the issue there?

@GuyTevet I only get TypeErrors, AttributeErrors (because I don't know what I'm doing), I just reshaped it (reshape(self.real_num_frames, 72), (reshape(self.real_num_frames, 3) to see if it's working.

[GuyTevet]

The shape of thetas is [24, 6, frames]. 6 stands for the 6d, you can look in our code or in ACTOR or MotionCLIP repos for references on how to use those functions.

[mmdrahmani]

Hi Thank you for the excellent work. I have tried the code and it works just fine. I even used the mesh object sequence in blender and that was also fine (really interesting results). I only had a basic question, regarding how to use the npy file in the blender SMPL or SMPL-X add on in order to create animations (similar to animations for the paper). I think this is mostly my fault because I know little about blender. I appreciate any information, or link to blender tutorials. I watched several SMPL/X blender tutorials on youtube but none of them addressed my problem. All the bests Mohammad

[mmdrahmani]

lso GPL3 license

Thanks for the code. This is amazing! How should I used SMPL neutral body instead of armature? Thanks

[alextitonis]

@amb Tried to use the script and exported the fbx, but the result was an orb with spykes around moving Is there a different way to export it? bpy.ops.export_scene.fbx(filepath="./tata.fbx", use_selection=True, add_leaf_bones=False)

[jaykup1]

Conversion of data to .pkl file for 'softcat477/SMPL-to-FBX/'

@jjp9624022 @TREE-Ind @Tony2371 @amb @mmdrahmani @alextitonis tell me if there are any problems, thanks

[alextitonis]

It works like charm awesome @jaykup1

[TREE-Ind]

Getting the FBX python SDK setup correctly was a pain but can also confirm the conversion works for us as well.

[mmdrahmani]

Conversion of data to .pkl file for 'softcat477/SMPL-to-FBX/'

@jjp9624022 @TREE-Ind @Tony2371 @amb @mmdrahmani @alextitonis tell me if there are any problems, thanks

This is cool! Thank you. I ran the code and converted and saved the data into a pickle file. But I am still not sure how to use this pickle file in Blender. I have SMPL and SMPL-X addon installed on blender, but I don't see how to import animations saved in the pickle file. Thank you for your support.

[jaykup1]

@mmdrahmani use the pkl file with this https://github.com/softcat477/SMPL-to-Fbx

Change 'FbxTime.eFrames60's in SMPL-to-FBX/FbxReadWriter.py to 'FbxTime.eFrames30'

Known issues: Glitches below 30 fps exports in 'softcat477/SMPL-to-FBX/' (20 fps expected)

[TREE-Ind]

https://user-images.githubusercontent.com/30479526/195868811-80a23002-9a77-4b22-81ef-233d0c6621f8.mp4

Importing motion diffusion animations into UE5 at runtime now thx to the conversion code :D

[jaykup1]

Importing motion diffusion animations into UE5 at runtime now thx to the conversion code :D

you can half the time of export by reducing 'self.num_smplify_iters = 150' in simplify_loc2rot.py to 50. (30 seemed fine), I will probably make a simple gradio gui and push it.