How to train hrnet_w48 with DeepposeRegression process

[tiger990111]

trafficstars

Dear, I am now trying to fine-tune model of 2d hand pose. However, I found that mmpose didn't train too much deeppose 2d hand pose. I want to use Hrnet_W48 to train in the deeppose regression way. I modify the configure of mmpose\configs\wholebody\2d_kpt_sview_rgb_img\topdown_heatmap\coco-wholebody\hrnet_w48_coco_wholebody_384x288_dark_plus.py

What i have done is :

1. I modify the model dict with adding the neck and change the keypoint_head and test_cfg
2. I modify the train_pipeline with dict(type='TopDownGenerateTarget') -->dict(type='TopDownGenerateTargetRegression')

My modified config is as below:

_base_ = [
    '../../../../_base_/default_runtime.py',
    '../../../../_base_/datasets/coco_wholebody_hand.py'
]
# use the pre-trained model

evaluation = dict(interval=10, metric=['PCK', 'AUC', 'EPE'], save_best='AUC')

optimizer = dict(
    type='Adam',
    lr=1e-3, #5e-4
)
optimizer_config = dict(grad_clip=None)
# learning policy
lr_config = dict(
    policy='step',
    warmup='linear',
    warmup_iters=500,
    warmup_ratio=0.001,
    step=[170, 200])
total_epochs = 30 #210
log_config = dict(
    interval=10,
    hooks=[
        dict(type='TextLoggerHook'),
        # dict(type='TensorboardLoggerHook')
    ])

channel_cfg = dict(
    num_output_channels=21,
    dataset_joints=21,
    dataset_channel=[
        [
            0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
            19, 20
        ],
    ],
    inference_channel=[
        0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
        20
    ])

# model settings
model = dict(
    type='TopDown',
    pretrained=None,
    backbone=dict(
        type='HRNet',
        in_channels=3,
        extra=dict(
            stage1=dict(
                num_modules=1,
                num_branches=1,
                block='BOTTLENECK',
                num_blocks=(4, ),
                num_channels=(64, )),
            stage2=dict(
                num_modules=1,
                num_branches=2,
                block='BASIC',
                num_blocks=(4, 4),
                num_channels=(48, 96)),
            stage3=dict(
                num_modules=4,
                num_branches=3,
                block='BASIC',
                num_blocks=(4, 4, 4),
                num_channels=(48, 96, 192)),
            stage4=dict(
                num_modules=3,
                num_branches=4,
                block='BASIC',
                num_blocks=(4, 4, 4, 4),
                num_channels=(48, 96, 192, 384))),
    ),
    neck=dict(type='GlobalAveragePooling'),
    keypoint_head=dict(
        type='DeepposeRegressionHead',
        in_channels=2048,
        num_joints=channel_cfg['num_output_channels'],
        loss_keypoint=dict(type='SmoothL1Loss', use_target_weight=True)),
    train_cfg=dict(),
    test_cfg=dict(
        flip_test=True,
        post_process='default',
        shift_heatmap=True,
        modulate_kernel=11))

data_cfg = dict(
    image_size=[256, 256],
    heatmap_size=[64, 64],
    num_output_channels=channel_cfg['num_output_channels'],
    num_joints=channel_cfg['dataset_joints'],
    dataset_channel=channel_cfg['dataset_channel'],
    inference_channel=channel_cfg['inference_channel'])

train_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='TopDownRandomFlip', flip_prob=0.5),

    dict(type='TopDownRandomResize', size_range=[64, 128], resize_prob=0.3), # Perform random resize
    dict(type='TopDownRandomBlur', kernel_size=5, blur_prob=0.5), # Perform random blur

    dict(
        type='TopDownGetRandomScaleRotation', rot_factor=90, scale_factor=0.3),
    dict(type='TopDownAffine'),
    dict(type='ToTensor'),
    dict(
        type='NormalizeTensor',
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225]),
    dict(type='TopDownGenerateTargetRegression'),
    # dict(type='TopDownGenerateTarget', sigma=2, unbiased_encoding=True),
    dict(
        type='Collect',
        keys=['img', 'target', 'target_weight'],
        meta_keys=[
            'image_file', 'joints_3d', 'joints_3d_visible', 'center', 'scale',
            'rotation', 'flip_pairs'
        ]),
]

val_pipeline = [
    dict(type='LoadImageFromFile'),
    dict(type='TopDownAffine'),
    dict(type='ToTensor'),
    dict(
        type='NormalizeTensor',
        mean=[0.485, 0.456, 0.406],
        std=[0.229, 0.224, 0.225]),
    dict(
        type='Collect',
        keys=['img'],
        meta_keys=['image_file', 'center', 'scale', 'rotation', 'flip_pairs']),
]

test_pipeline = val_pipeline

data_root = ['/data/public_data/pose/hand2d/coco']
data = dict(
    samples_per_gpu=64,
    workers_per_gpu=2,
    val_dataloader=dict(samples_per_gpu=32),
    test_dataloader=dict(samples_per_gpu=32),
    train=[
        dict(
            type='HandCocoWholeBodyDataset',
            ann_file=f'{data_root[0]}/annotations/coco_wholebody_train_v1.0.json',
            img_prefix=f'{data_root[0]}/train2017/',
            data_cfg=data_cfg,
            pipeline=train_pipeline,
            dataset_info={{_base_.dataset_info}}
        )
    ],

    val=dict(
        type='HandCocoWholeBodyDataset',
        ann_file=f'{data_root[0]}/annotations/coco_wholebody_val_v1.0.json',
        img_prefix=f'{data_root[0]}/val2017/',
        data_cfg=data_cfg,
        pipeline=val_pipeline,
        dataset_info={{_base_.dataset_info}}
    ),
    test=dict(
        type='HandCocoWholeBodyDataset',
        ann_file=f'{data_root[0]}/annotations/coco_wholebody_val_v1.0.json',
        img_prefix=f'{data_root[0]}/val2017/',
        data_cfg=data_cfg,
        pipeline=val_pipeline,
        dataset_info={{_base_.dataset_info}}
    ),
)

ps: actually I am quite familiar with the structure of hrnet and the difference between deeppose processing and heatmap processing. So what i try to do is mimic the config of both of them. I have no idea that those exact meaning in code. Grateful for your help!

[ly015]

Your steps are correct. Are you able to train the model with the modified config?