sign
[BUG] The Mixamorg Avatar stays still in it's T pose.
Current Behavior
The 3d avatar does not move for any input, it stays in its t pose.
Expected Behavior
The mixamorg avatar is supposed to sign.
Steps To Reproduce
No response
Example flow
paste your flow here
Environment
- Device: Macbook
- OS: MacOS
- Browser:Chrome
- Version: (eg. 13; 73)
- Other info: (eg. display resolution, ease-of-access settings)
Additional context
No response
Given the poor performance in https://www.youtube.com/watch?v=TyJuU9_GOaw We did not further implement this avatar movement in the translation app, but you can test it out in the Playground https://sign.mt/playground
If we were to have a better model, we would implement it.
Hi, I was wondering why we can't directly map the 3d keypoints obtained as pose file into the bone structure of the mixamorig avatar. It will be great if you can share the Structure of the pose file
Hi, I was wondering why we can't directly map the 3d keypoints obtained as pose file into the bone structure of the mixamorig avatar. It will be great if you can share the Structure of the pose file
On Sun, Apr 20, 2025 at 1:16 PM Amit Moryossef @.***> wrote:
Given the poor performance in https://www.youtube.com/watch?v=TyJuU9_GOaw We did not further implement this avatar movement in the translation app, but you can test it out in the Playground https://sign.mt/playground
If we were to have a better model, we would implement it.
— Reply to this email directly, view it on GitHub https://github.com/sign/translate/issues/231#issuecomment-2817046610, or unsubscribe https://github.com/notifications/unsubscribe-auth/A4OQDZNXKLG6M4WZF4BM3GD22NGFDAVCNFSM6AAAAAB3PELGVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMJXGA2DMNRRGA . You are receiving this because you authored the thread.Message ID: @.***> AmitMY left a comment (sign/translate#231) https://github.com/sign/translate/issues/231#issuecomment-2817046610
Given the poor performance in https://www.youtube.com/watch?v=TyJuU9_GOaw We did not further implement this avatar movement in the translation app, but you can test it out in the Playground https://sign.mt/playground
If we were to have a better model, we would implement it.
— Reply to this email directly, view it on GitHub https://github.com/sign/translate/issues/231#issuecomment-2817046610, or unsubscribe https://github.com/notifications/unsubscribe-auth/A4OQDZNXKLG6M4WZF4BM3GD22NGFDAVCNFSM6AAAAAB3PELGVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMJXGA2DMNRRGA . You are receiving this because you authored the thread.Message ID: @.***>
waiting for your response
On Sun, Apr 20, 2025 at 9:21 PM Usha nandhini @.***> wrote:
Hi, I was wondering why we can't directly map the 3d keypoints obtained as pose file into the bone structure of the mixamorig avatar. It will be great if you can share the Structure of the pose file
On Sun, Apr 20, 2025 at 1:16 PM Amit Moryossef @.***> wrote:
Given the poor performance in https://www.youtube.com/watch?v=TyJuU9_GOaw We did not further implement this avatar movement in the translation app, but you can test it out in the Playground https://sign.mt/playground
If we were to have a better model, we would implement it.
— Reply to this email directly, view it on GitHub https://github.com/sign/translate/issues/231#issuecomment-2817046610, or unsubscribe https://github.com/notifications/unsubscribe-auth/A4OQDZNXKLG6M4WZF4BM3GD22NGFDAVCNFSM6AAAAAB3PELGVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMJXGA2DMNRRGA . You are receiving this because you authored the thread.Message ID: @.***> AmitMY left a comment (sign/translate#231) https://github.com/sign/translate/issues/231#issuecomment-2817046610
Given the poor performance in https://www.youtube.com/watch?v=TyJuU9_GOaw We did not further implement this avatar movement in the translation app, but you can test it out in the Playground https://sign.mt/playground
If we were to have a better model, we would implement it.
— Reply to this email directly, view it on GitHub https://github.com/sign/translate/issues/231#issuecomment-2817046610, or unsubscribe https://github.com/notifications/unsubscribe-auth/A4OQDZNXKLG6M4WZF4BM3GD22NGFDAVCNFSM6AAAAAB3PELGVCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDQMJXGA2DMNRRGA . You are receiving this because you authored the thread.Message ID: @.***>
The .pose file has a .header (you can check out the format here https://github.com/sign-language-processing/pose)
That gives you points and their 3D positions.
In theory, what you are saying is possible. I have seen people map the keypoints to a 3D mesh, but the main problem is that some specific interactions (for example, the letter M in ASL) is not fully correctly represented in MediaPipe. For that, a model could "recover" from the errors - if it sees something strange at this or that pattern, it can learn how to recover.
I did some attempts at looking at the 3D positioning of the predicted pose, you can look at these videos:
https://github.com/user-attachments/assets/878ee0ef-e4d3-4dad-ace4-aab6fcb5e17d
https://github.com/user-attachments/assets/eed641e9-87fe-4474-85d5-9ebe0c5977c2
hi, I am adding to the discussion. I created a project for my work that moves an avatar in real time by capturing mediapipe from pc camera. It works really well and I can tell you that with mixamo this cannot be done at all. To create an avatar that correctly interprets the pose, face and hands keypoints of mediapipe you need to work with blender (for example) and do the correct rigging of all the bones, create the meshes and determine what influences the meshes have based on the rotation or displacement of the individual bones. Mixamo's “puppets” do not support all this. The princiapal problems established the avatar with blender and realtime motion mapping, or passing a pre-recorded motion of keypoints mediapipe are:
- jitter: can be mitigated with smoothNET by disabling that of mediapipe poses if we prerecord the movements and give them in batch to the avatar
- Z-axis of the pose: established a fixed front view and optionally blocking the rotation of the pelvis, the Z-axis shoulder joint is still not properly evaluated by mediapipe because not one camera you don't have depth and mediapipe has a 2D view of the captured frame. I am working on this since I want to use only one camera, e.g., the phone camera. Finally keep in mind that threejs is webgl and not opengl so it does not take full advantage of avatar rendering (glb/gltf). If you want avatar beautiful as on blender that are GPU you have to write native code that takes advantage of hardware resources.
Very cool @marsiliflavio Do you have a demo to try?
Want to integrate it with the app? If so, I think all we need is a GLTF file for the avatar with all the bones etc (and make sure it loads with model-viewer) then your mapping function.
Can the email [email protected] be used to talk to you? I can't find your email in the profile. your work is super. i am part of project called HandySigns for LIS (italian sign language) translation and mostly app/web and avatar development
you can reach out at [email protected], however, any message related to this project, as long as not containing some secret information, i'd rather manage in github.