keypoint results have a problem.

[shinysoo]

trafficstars

Hello, I found some problem in the keypoint results. I used the 2d human wholebody model and I saved keypoint results of left elbow per each frame. The results are as follows:

[[533.2927659913354, 242.27483407574857],
 [542.8659101753923, 173.42588643023555], # weird frame
 [529.3694469490923, 238.64215592714027],
 [529.1746711316422, 238.54780465158785],
 [533.1372074853036, 239.98222345573595],
 [535.1726164480035, 241.3417378658778],
 [531.2928084654673, 237.34142811944912],
 [536.1837568412957, 238.32033314843045],
 [533.9527751089408, 240.7851721199404],
 [536.6088303611881, 198.45103853302624], # weird frame
 [533.1080307119312, 240.92316460014501],
 [529.876545164566, 239.23669665380658],
 [532.8686809404041, 241.07155382764904],
 [533.7616996957804, 240.52373577156686],
 [536.5621450167284, 239.8132072718472],
 [543.4726603540136, 174.64059555382607], # weird frame
 [534.9431595854549, 241.30164728193876],
		.
		.
		.
]

My video is about moving elbow smoothly, but the # weird frame's value changes dramatically. Not only the elbow, but also other moving joints have the same phenomenon. But the motionless joints don't show that. Also the visualization result is good. It didn't show any outliers in the visualization.

Is it a normal phenomenon? or Is it a problem that needs to be fixed?

[jin-s13]

Since the model is frame-based (without temporal info), it is normal that in some frames the value changes dramatically. Simple post-processing (smooth) will solve this problem.

[shinysoo]

Then I should change it to the average value of the front and back frames. Thank you!

[shinysoo]

@jin-s13 Um.. I visualized the kepoint results of left elbow's y as a graph. I expected a soft wave pattern because a person in my video moves his arm up and down . But I found that so many noises exist. The code and graph are as follows:

python demo/topdown_demo_with_mmdet.py \
    demo/mmdetection_cfg/faster_rcnn_r50_fpn_coco.py \
    https://download.openmmlab.com/mmdetection/v2.0/faster_rcnn/faster_rcnn_r50_fpn_1x_coco/faster_rcnn_r50_fpn_1x_coco_20200130-047c8118.pth \
    configs/wholebody_2d_keypoint/topdown_heatmap/coco-wholebody/td-hm_hrnet-w48_dark-8xb32-210e_coco-wholebody-384x288.py \
    https://download.openmmlab.com/mmpose/top_down/hrnet/hrnet_w48_coco_wholebody_384x288_dark-f5726563_20200918.pth \
    --input my_video \
    --output-root vis_results  \
    --show --save-predictions

path = 'vis_results/results_my_video.json'
with open(path, 'r') as f:
    json_result = json.load(f)

left_elbow_y = []
for i in range(len(json_result['instance_info'])):
    left_elbow_y.append(json_result['instance_info'][i]['instances'][0]['keypoints'][7][1])

plt.style.use('seaborn')
plt.rcParams['figure.figsize'] = (15, 4)
x = np.arange(len(left_hip_y))
fig, ax1 = plt.subplots()
ax1.plot(x, left_hip_y, color='blue')

The video size is (960, 540).

The interesting thing was that the shoulder and wrist showed the same pattern. However, since the shoulder has small movement, only about 10 pixels of error are seen.

[y of left shoulder]

[y of left wrist]

What mistake did I make? or is it model's problem? Could you check and test my code and model?