MB-Lab
MB-Lab copied to clipboard
animate1978
Request: Facial Motion capture: Option to match iOS Shape-key naming scheme.
This is a feature request that will enhance MBlab by making it easier to use iOS facial mocap data with MBlab. Here is a link to Apples blend shapes: https://developer.apple.com/documentation/arkit/arfaceanchor/blendshapelocation
A popular app that records Facial Mocap data on iOS includes Facecap: http://www.bannaflak.com/face-cap/ Facecap exports it's data as FBX. To use in blender first convert from ascii fbx to binary fbx using Autodesk fbx converter. Importing the FBX into blender gives you a facial mesh, shapekeys with animation, and then a few empties for head rotation / movement and 2 that controls the eyes in some way. Demo FBX files (Needs to be converted using Autodesk FBX Converter): https://drive.google.com/file/d/1Xs2IQzKhoV-qpRE2ymhanaxnnYaH1EA0/view?usp=sharing
The best type of solution I can think of, would be similar to the BVH import function, where you click a button to import the FBX w/ the facial motion capture data, and then from there it applies to the MBlab character mesh.
An alternative solution, would be an option during character creation, to follow the iOS shape key naming scheme, then you can copy and paste the animation values as described in: https://youtu.be/ZYrkKi7_qdY?t=223
There have also been people who have figured out ways to send live data via osc, however its overly complicated / not in scope of MBlab.
This could also lead the way to FBX support for body motion capture, such as from Mixamo. where currently I have to export my character model, import into Mixamo, then import the FBX into blender, this then looses the facial shape-keys, to preserve those: export the imported mixamo animated character as BVH, to then import using MBlab's BVH option onto my character
With half the world having an iphone capable of recording facial mocap data, this method of facial mocap will easily become a standard in the industry, with solutions such as iclone already developing the software to do this.
These are the shape key names from the Facecap fbx model:
Facecap
Basis browInnerUp browDown_L browDown_R browOuterUp_L browOuterUp_R eyeLookUp_L eyeLookUp_R eyeLookDown_L eyeLookDown_R eyeLookIn_L eyeLookin_R eyeLookout_L eyeLookout_R eyeBIink_L eyeBIink_R eyeSquint_L eyeSquint_R eyeWide_L eyeWide_R cheekPuff cheekSquint_L cheekSquint_R nosesneer_L nosesneer_R jawOpen jawForward jawLeft jawRight mouthFunneI mouthPucker mouthLeft mouthRight mouthRoIIUpper mouthRoIILower mouthshrugUpper mouthShrugLower mouthCIose mouthSmiIe_L mouthSmiIe_R mouthFrown_L mouthFrown_R mouthDimpIe_L mouthDimpIe_R mouthupperUp_L mouthUpperUp_R mouthLowerDown_L mouthLowerDown_R mouthpress_L mouthPress_R mouthStretch_L mouthStretch_R
I think with the new JSON creation / editing tools that are being developed I think it may be possible in the future.
When I am not sure a lot of this is brand new and we are just barely scratching the surface of this possibility
Sounds exciting! This shape key feature would definitely elevate MB-Lab to a new level.
The OP already gave some very good points. Just to reiterate:
The Face Cap app gives really good facial capture results with little effort. Being able to directly pipe them to a MB-lab character would be an extremely valuable option. I'd be happy with an initial implementation just having a option to change the shape key names during the creation process to match those of IOS.
Thank you for all the hard work!
If I had a list of what the current name is and the change to names, it wouldn't be hard to rename them, but first I would have to know what names are changed. Ex. Old name -> New name Old name -> New name ...
On Thu, Feb 13, 2020, 2:36 PM Jaydead [email protected] wrote:
Sounds exciting! This shape key feature would definitely elevate MB-Lab to a new level.
The OP already gave some very good points. Just to reiterate:
The Face Cap app gives really good facial capture results with little effort. Being able to directly pipe them to a MB-lab character would be an extremely valuable option. I'd be happy with an initial implementation just having a option to change the shape key names during the creation process to match those of IOS.
Thank you for all the hard work!
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/animate1978/MB-Lab/issues/219?email_source=notifications&email_token=AKYHC3VPKY4OVKDZ2HGQ6RDRCWONBA5CNFSM4KCRGUTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELWKQZI#issuecomment-585934949, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKYHC3VKWEPRFY5I3UJ26M3RCWONBANCNFSM4KCRGUTA .
If I had a list of what the current name is and the change to names, it wouldn't be hard to rename them, but first I would have to know what names are changed. Ex. Old name -> New name Old name -> New name ... … On Thu, Feb 13, 2020, 2:36 PM Jaydead @.***> wrote: Sounds exciting! This shape key feature would definitely elevate MB-Lab to a new level. The OP already gave some very good points. Just to reiterate: The Face Cap app gives really good facial capture results with little effort. Being able to directly pipe them to a MB-lab character would be an extremely valuable option. I'd be happy with an initial implementation just having a option to change the shape key names during the creation process to match those of IOS. Thank you for all the hard work! — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#219?email_source=notifications&email_token=AKYHC3VPKY4OVKDZ2HGQ6RDRCWONBA5CNFSM4KCRGUTKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOELWKQZI#issuecomment-585934949>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKYHC3VKWEPRFY5I3UJ26M3RCWONBANCNFSM4KCRGUTA .
It looks like one user at Blenderartists has done some more testing regarding this, and it might need more than just renaming. Apparently only a small part of the shape keys could be directly renamed.
There's a list in the post of shape key names for both systems.
Perhaps the shape keys could be revamped further down the line, or a new set could be added? Looks like there's a lot of redundant ones in the current system.
That was actually me too, that's just my username over there.
I ended up giving up on MBlab after spending a weekend of trying to match the two shape-key sets. Since being able to use the iphone as a facial mocap solution is a crucial part of my workflow, I ended up giving up and switching over to Mixamo Fuse. While the textures are lacking, and I'll lose the ability to create new characters in the fall, it is allowing me to go from character creation to animation very easily. Using animation nodes I was able to quickly transfer the animation key data between the facecap and fuse models.
I'm still crossing my fingers that MBlab will add some sort of functionality I can use out of the box. I miss certain features of MBlab such as the BVH motion capture import.
I also wanted to point out that there is a person (I think he is also developer for blender?), who is developing mocap both facial and full body using the iphone for blender: https://www.youtube.com/watch?v=ixfACC5nxwE&feature=emb_logo He is planning on his next release having compatibility with MakeHuman, I suggested that he also have compatibility with MBlab, since MBlab is new and has more potential.