VIBE
VIBE copied to clipboard
[BUG] Fix "can not training with 2d dataset PennAction and PoseTrack"
Thanks @ZhangHedongTimesand @SatoshiYamazaki for the discussion: https://github.com/mkocabas/VIBE/issues/154
I fixed these bugs and summarized here.
PoseTrack:
Replace
kp_2d = convert_kps(kp_2d, src=self.dataset_name, dst='spin')
with
kp_2d = np.copy(self.db['joints2D'][start_index:end_index+1])
PennAction: When I visualized the keypoints in feature_extractor during preprocessing datasets, I found the first keypoint appeared is left ankle while "rankle" is the first one in def get_common_joint_names() function. All the keypoints is converse. When authors preprocessed the pennaction dataset, they converted penn format to common (converse) first in penn_action_utils.py.
perm_idxs = get_perm_idxs('pennaction', 'common')
kp_2d[:, :, 0] = vid_dict['x']
kp_2d[:, :, 1] = vid_dict['y']
kp_2d[:, :, 2] = vid_dict['visibility']
kp_2d = kp_2d[:, perm_idxs, :]
But kp_2d = convert_kps(kp_2d, src=self.dataset_name, dst='spin') in dataset_2d.py will regard penn keypoint format as pennaction and convert them to spin format.
So the following two stages will solve these problems.
- Replace function 'def get_common_joint_names()' in lib/data_utils/kp_utils.py with
def get_common_joint_names():
return [
"lankle",
"lknee",
"lhip",
"rhip",
"rknee",
"rankle",
"lwrist",
"lelbow",
"lshoulder",
"rshoulder",
"relbow",
"rwrist",
"neck",
"headtop",
]
- Modify def get_single_item(self, index) in lib/dataset/dataset_2d.py as follow
if self.dataset_name != 'posetrack':
if self.dataset_name != 'pennaction':
kp_2d = convert_kps(kp_2d, src=self.dataset_name, dst='spin')
else:
kp_2d = convert_kps(kp_2d, src='common', dst='spin')
These problems may be caused by the inconsistency between the index chosen by the author during preprocessing and the final code given. So the problems may only on downloaded feature datasets. If you preprocess dataset by yourself, these problems may not arise.