video2bvh
video2bvh copied to clipboard
KevinLTT
Is it possible to use mediapipe for converting the pose to bvh?
I am currently looking into obtainig the pose from a video/image and then converting them into BVH format.I am looking at using Mediapipe for 3d world pose estimation,but it seems that the mediapipe skeleton and openpose skeleton are a bit different. for example the current mediapipe skeleton mapping that I am working on is as follows
self.keypoint2index = {
'mid_hip': 0,
'right_hip': 1,
'right_knee': 2,
'right_ankle': 3,
'right_foot_index': 4,
'left_hip': 5,
'left_knee': 6,
'left_ankle': 7,
'left_foot_index': 8,
'spine': 9,
'mid_shoulder': 10,
'left_shoulder': 11,
'left_elbow': 12,
'left_wrist': 13,
'right_shoulder': 14,
'right_elbow': 15,
'right_wrist': 16,
'right_foot_EndSite': -1,
'left_foot_EndSite': -1,
'left_wrist_EndSite': -1,
'right_wrist_EndSite': -1,
'mid_shoulder_EndSite': -1
}
I also noticed that when I view the bvh in blender,the axes need to be rotated by 360 degree to see the model in correct pose.In my code,the files bvh_helper,math3d are the same as yours.Only changes i did were in the cmu_skeleton before line 119.The changes are mentioned below
self.index2keypoint = {v: k for k, v in self.keypoint2index.items()}
self.keypoint_num = len(self.keypoint2index)
# Create the parent-children dictionary
self.children = {
'mid_hip': ['left_hip', 'spine', 'right_hip'],
'left_hip': ['left_knee'],
'left_knee': ['left_ankle'],
'left_ankle': ['left_foot_index'],
'left_foot_index': ['left_foot_EndSite'],
'left_foot_EndSite': [],
'spine': ['mid_shoulder'],
'mid_shoulder': ['left_shoulder', 'right_shoulder', 'mid_shoulder_EndSite'],
'mid_shoulder_EndSite': [],
'left_shoulder': ['left_elbow'],
'left_elbow': ['left_wrist'],
'left_wrist': ['left_wrist_EndSite'],
'left_wrist_EndSite': [],
'right_shoulder': ['right_elbow'],
'right_elbow': ['right_wrist'],
'right_wrist': ['right_wrist_EndSite'],
'right_wrist_EndSite': [],
'right_hip': ['right_knee'],
'right_knee': ['right_ankle'],
'right_ankle': ['right_foot_index'],
'right_foot_index': ['right_foot_EndSite'],
'right_foot_EndSite': []
}
self.parent = {self.root: None}
for parent, children in self.children.items():
for child in children:
self.parent[child] = parent
self.left_joints = [
joint for joint in self.keypoint2index
if 'left_' in joint
]
self.right_joints = [
joint for joint in self.keypoint2index
if 'right_' in joint
]
# Create T-pose and define the direction from parent to child joint
# What if the direction is not aligned with any axis????? I don't know, may be use sin, cos to represent it???????????????????
self.initial_directions = {
'mid_hip': [0, 0, 0],
'right_hip': [-1, 0, 0],
'right_knee': [0, 0, -1],
'right_ankle': [0, 0, -1],
'right_foot_index': [0, -1, 0],
'right_foot_EndSite': [0, -1, 0],
'left_hip': [1, 0, 0],
'left_knee': [0, 0, -1],
'left_ankle': [0, 0, -1],
'left_foot_index': [0, -1, 0],
'left_foot_EndSite': [0, -1, 0],
'spine': [0, 0, 1],
'mid_shoulder': [0, 0, 1],
'mid_shoulder_EndSite': [0, 0, 1],
'left_shoulder': [1, 0, 0],
'left_elbow': [1, 0, 0],
'left_wrist': [1, 0, 0],
'left_wrist_EndSite': [1, 0, 0],
'right_shoulder': [-1, 0, 0],
'right_elbow': [-1, 0, 0],
'right_wrist': [-1, 0, 0],
'right_wrist_EndSite': [-1, 0, 0]
}
I would appreciate it if you could suggest me some pointers on what can be done to correct the final pose in my case.
I am currently looking into obtainig the pose from a video/image and then converting them into BVH format.I am looking at using Mediapipe for 3d world pose estimation
Hey, were you able to make any progress on this? I'm attempting the same thing, or rather I'm attempting to turn video into some generic and widely used format representing posture/animation data, and I happened to land on mediapipe and BVH.
I'd be interested to hear if you made any useful findings, perhaps we can share some experiences.
I am currently looking into obtainig the pose from a video/image and then converting them into BVH format.I am looking at using Mediapipe for 3d world pose estimation,but it seems that the mediapipe skeleton and openpose skeleton are a bit different. for example the current mediapipe skeleton mapping that I am working on is as follows
self.keypoint2index = { 'mid_hip': 0, 'right_hip': 1, 'right_knee': 2, 'right_ankle': 3, 'right_foot_index': 4, 'left_hip': 5, 'left_knee': 6, 'left_ankle': 7, 'left_foot_index': 8, 'spine': 9, 'mid_shoulder': 10, 'left_shoulder': 11, 'left_elbow': 12, 'left_wrist': 13, 'right_shoulder': 14, 'right_elbow': 15, 'right_wrist': 16, 'right_foot_EndSite': -1, 'left_foot_EndSite': -1, 'left_wrist_EndSite': -1, 'right_wrist_EndSite': -1, 'mid_shoulder_EndSite': -1 }I also noticed that when I view the bvh in blender,the axes need to be rotated by 360 degree to see the model in correct pose.In my code,the files bvh_helper,math3d are the same as yours.Only changes i did were in the cmu_skeleton before line 119.The changes are mentioned below
self.index2keypoint = {v: k for k, v in self.keypoint2index.items()} self.keypoint_num = len(self.keypoint2index) # Create the parent-children dictionary self.children = { 'mid_hip': ['left_hip', 'spine', 'right_hip'], 'left_hip': ['left_knee'], 'left_knee': ['left_ankle'], 'left_ankle': ['left_foot_index'], 'left_foot_index': ['left_foot_EndSite'], 'left_foot_EndSite': [], 'spine': ['mid_shoulder'], 'mid_shoulder': ['left_shoulder', 'right_shoulder', 'mid_shoulder_EndSite'], 'mid_shoulder_EndSite': [], 'left_shoulder': ['left_elbow'], 'left_elbow': ['left_wrist'], 'left_wrist': ['left_wrist_EndSite'], 'left_wrist_EndSite': [], 'right_shoulder': ['right_elbow'], 'right_elbow': ['right_wrist'], 'right_wrist': ['right_wrist_EndSite'], 'right_wrist_EndSite': [], 'right_hip': ['right_knee'], 'right_knee': ['right_ankle'], 'right_ankle': ['right_foot_index'], 'right_foot_index': ['right_foot_EndSite'], 'right_foot_EndSite': [] } self.parent = {self.root: None} for parent, children in self.children.items(): for child in children: self.parent[child] = parent self.left_joints = [ joint for joint in self.keypoint2index if 'left_' in joint ] self.right_joints = [ joint for joint in self.keypoint2index if 'right_' in joint ] # Create T-pose and define the direction from parent to child joint # What if the direction is not aligned with any axis????? I don't know, may be use sin, cos to represent it??????????????????? self.initial_directions = { 'mid_hip': [0, 0, 0], 'right_hip': [-1, 0, 0], 'right_knee': [0, 0, -1], 'right_ankle': [0, 0, -1], 'right_foot_index': [0, -1, 0], 'right_foot_EndSite': [0, -1, 0], 'left_hip': [1, 0, 0], 'left_knee': [0, 0, -1], 'left_ankle': [0, 0, -1], 'left_foot_index': [0, -1, 0], 'left_foot_EndSite': [0, -1, 0], 'spine': [0, 0, 1], 'mid_shoulder': [0, 0, 1], 'mid_shoulder_EndSite': [0, 0, 1], 'left_shoulder': [1, 0, 0], 'left_elbow': [1, 0, 0], 'left_wrist': [1, 0, 0], 'left_wrist_EndSite': [1, 0, 0], 'right_shoulder': [-1, 0, 0], 'right_elbow': [-1, 0, 0], 'right_wrist': [-1, 0, 0], 'right_wrist_EndSite': [-1, 0, 0] }I would appreciate it if you could suggest me some pointers on what can be done to correct the final pose in my case.
hi,have you solved this problem?thanks
