Different results when dataset channel is changed

[kinglr]

trafficstars

Hello, thank you for the great toolbox! I am getting two different results when I use

channel_cfg = dict(
    num_output_channels=17+14,
    dataset_joints=14,
    dataset_channel=[0,1,2,3,4,5,6,7,8,9,10,11,12,13],
    inference_channel=[0,1,2,3,4,5,6,7,8,9,10,11,12,13]
)

and

channel_cfg = dict(
    num_output_channels=17+14,
    dataset_joints=14,
    dataset_channel=[17,18,19,20,21,22,23,24,25,26,27,28,29,30],
    inference_channel=[17,18,19,20,21,22,23,24,25,26,27,28,29,30]
)

In just the first 10 epochs, the validation AP for the first config is around 0.4 and 0.16 for the second one.

I am using FuseDataset to insert the joints at its respective dataset channels as follows:

fuse_target = np.zeros((num_output_channels, target.shape[1], target.shape[2]), dtype=np.float32)
fuse_target[dataset_channel] = target
# same for target_weight

and I am selecting the inference channels during evaluation in top_down_coco_dataset.py as follows: preds = result['preds'][:,inference_channel]

I don't understand where the issue is coming from. Is there any reason why this would happen?

Thank you!

[Ben-Louis]

Hi, thanks for using MMPose. Have you modified the dataset metainfo config files (e.g. configs/_base_/datasets/coco.py)? Joints' symmetric relationship and sigmas, which will influence the evaluation process, are defined in those config files. The indices of each joint in keypoint_info should change accordingly when the inference_channel and dataset_channel change.

[kinglr]

Hi @Ben-Louis, thank you for the quick feedback! I tried changing the keypoint_info keys to be similar with the dataset_channel indices, but it raised other issues. I would be very grateful if you can give me an example for:

channel_cfg = dict(
    num_output_channels=17+14,
    dataset_joints=14,
    dataset_channel=[17,18,19,20,21,22,23,24,25,26,27,28,29,30],
    inference_channel=[17,18,19,20,21,22,23,24,25,26,27,28,29,30]
)

[Ben-Louis]

Suppose the original keypoint_info is

keypoint_info={
    0:
    dict(name='nose', id=0, color=[51, 153, 255], type='upper', swap=''),
    1:
    dict(
        name='left_eye',
        id=1,
        color=[51, 153, 255],
        type='upper',
        swap='right_eye'),
    ...
    13:
    dict(
        name='left_knee',
        id=13,
        color=[0, 255, 0],
        type='lower',
        swap='right_knee'),
}

If you want to change the indices of those keypoints to 17-30, you could try to modify the keypoint_info to:

keypoint_info={
    0: dict(name='kpt0', id=0),
    1: dict(name='kpt1', id=1),
    ...
    16: dict(name='kpt16', id=16),
    17: dict(name='nose', id=17, color=[51, 153, 255], type='upper', swap=''),
    18:
    dict(
        name='left_eye',
        id=18,
        color=[51, 153, 255],
        type='upper',
        swap='right_eye'),
    ...
    30:dict(
        name='left_knee',
        id=30,
        color=[0, 255, 0],
        type='lower',
        swap='right_knee'),
}

[kinglr]

Thank you, @Ben-Louis! I have tried that and the first issue was:

AssertionError: File "../mmpose/datasets/datasets/base/kpt_2d_sview_rgb_img_top_down_dataset.py", line 74, in __init__
    assert self.ann_info['num_joints'] == dataset_info.keypoint_num

I commented that out. Then, the next issue was:

mmpose/core/post_processing/post_transforms.py", line 41, in fliplr_joints
    joints_3d_flipped[left, :] = joints_3d[right, :]
IndexError: index 14 is out of bounds for axis 0 with size 14
raise self.exc_type(msg)

With my initial experiment, I don't understand why changing the dataset_channel raises these issues as far as I am using it at the end of the pipeline. The number of joints is 14 until the last transformation (FuseDataset). The number of joints is also 14 after selecting the inference channel in the evaluate function of top_down_coco_dataset.py. Can you point me to functions where the reason for these issues could be? Thank you!