How to use Effect Renderer?

[MrMechanical]

This is definitely one of the coolest Unity plugins of all time. I am curious if the MediaPipe "Effect Renderer" is supported and how to use it with the Unity MediaPipe Plugin. I am attempting to create ar face filters using Unity Media Pipe Plugin. Thank you so much for reading, any and all help is greatly appreciated.

Kind Regards,

Mr-Mechanical

[MrMechanical]

Dear @homuler

I am planning on using the Face Geometry module to create a version of the "Effect Renderer" for Unity where Unity does the 3D rendering instead of the OpenGL solution provided by Google's mediapipe. I am unsure of how to run the Face Geometry module in the Unity MediaPipe plugin and retrieve the Face pose transformation matrix and the 3D Face Mesh (I am new to mediapipe).

Thank you so much for your help,

Mr-Mecahnical.

[homuler]

Currently, this plugin cannot handle 3D Face Mesh (to be precise, Packet<Mesh3d> class is not implemented). I'm going to implement a sample to run Face Effect solution this month.

[MrMechanical]

Thank you so much for your response. Do you think it would be possible for you to implement the Face Effect sample utilizing FaceLandmarks + FaceGeometry in order to render the output mesh through Unity (using FaceGeometry output data to render mesh/position object in Unity)? This would allow higher quality rendering than FaceEffect's simple OpenGL renderer and might be much simpler to implement (just adding FaceGeometry calculator node to FaceLandmark demo). Thank you for sharing this truly incredible one-of-a-kind Unity plugin with the community. I am very grateful for your amazing plugin.

Mr-Mechanical

[Jatin-Pawar]

Hey, Any progress on this end? @MrMechanical Did you try to make ar face filters using MediaPipe? Were you able make any progress?

Also, @homuler as of now, do we get the 3D data (like position (x,y,z) and rotation (x,y,z))? If yes, how can we access it. If no, when can we expect to get this 3D data?

Regards ~jk!ng

[MrMechanical]

Thank you so much for your interest. I'm going to add AR face filters directly rendered in Unity once the Face Effect sample is ready for Unity media pipe.

[mikedreambooth]

Hey @homuler. Thanks for your work on this! any update on the implementation of Packet<Mesh3d> ?

Mike

[homuler]

I'm sorry that I've not started it yet. Many issues are created every week and I'm trying to solve them radically and make them not being created from the beginning.

I expect I can start to tackle this issue at the end of the month.

[prisonerjohn]

Hello @homuler just wondering if you were able to get this going yet? Is there anything we can do to help?

[am-badshah]

@homuler any update on the Packet<Mesh3d> sample.

[wizVR-zhangjun]

@homuler I realized the face real-time drive, but I do not know how to take a glasses on the face, ask the teacher to give some advice

https://user-images.githubusercontent.com/60952586/187395131-5a69a430-f705-4d1d-bb83-711b84d43ba8.mp4 I don't know why this face why are the eyes closed

[MahdiHabibiXr]

Hey, Is the 3d facemesh available now? 😁