Im running mediapipe on windows10 with the python api Im using a logitech webcam for real time input. I am using the python examples provided in the mediapipe docs.
The solutions.hands is successfully tracking hands consistently. Unfortunately there is a very noticeable amount of landmark jitter even when the hands are not moving.

In an attempt to resolve the jitter I switched to the solutions.holistic which has a SMOOTH_LANDMARKS option. Using the holistic solution did reduce the landmark jitter but the hand tracking failed more often than the hands solution.

This post is a feature request to add SMOOTH_LANDMARKS to solutions.hands

Hi @willanie , SMOOTH_LANDMARKS, this line https://github.com/google/mediapipe/blob/710fb3de58dc10b7bc75f9cb758300c9016a5e4f/mediapipe/modules/pose_landmark/pose_landmark_model_loader.pbtxt#L21 specifies the default selection in the SwitchContainer. A value of 1 points to the middle one (0, 1, 2), which is "mediapipe/modules/pose_landmark/pose_landmark_full.tflite

Hi @sureshdagooglecom, I took a look at the link you supplied. Could you please explain how that block of code is related to SMOOTH_LANDMARKS

here is my hands init parameter settings

with mp_hands.Hands( model_complexity=0, static_image_mode=False, max_num_hands=2, min_detection_confidence=0.5, min_tracking_confidence=0.5) as hands:

when I add this to the hands init SMOOTH_LANDMARKS=True,

I get this python error

TypeError: Hands.init() got an unexpected keyword argument 'SMOOTH_LANDMARKS'

I took a look at the code in the link below and it has these comments but I dont understand how to engage filtering in the hands solution

MediaPipe graph to filter landmarks temporally (across packets with

incremental timestamps) to reduce jitter.

https://github.com/google/mediapipe/blob/710fb3de58dc10b7bc75f9cb758300c9016a5e4f/mediapipe/modules/pose_landmark/pose_landmark_filtering.pbtxt

Hi @willanie , The Hand's init method now has 5 params (see https://github.com/google/mediapipe/blob/master/mediapipe/python/solutions/hands.py#L89) and the third parameter is model_complexity, which is an int. Please modify your code accordingly.

Hi @sureshdagooglecom ,

I'm using the following solutions.hands paramaters with mp_hands.Hands( model_complexity=1, static_image_mode=False, max_num_hands=2, min_detection_confidence=0.5, min_tracking_confidence=0.5) as hands:

The hand and fingers track very well. Setting model_complexity to 1 does seem to reduce the jitter slightly. But there is still quite a lot of landmark jitter when the hand is moving very little, as you can see in this movie.

https://user-images.githubusercontent.com/4933879/169457295-48cfe6e1-2c1c-4a05-8c02-e61ddec3515c.mp4

I've begun investigating implementing one euro temporal filtering but it introduces lag issues.

Hi @sureshdagooglecom , Solutions.hands offers impressively reliable consistency in tracking hand and finger motion. Unfortunately, the amplitude of landmark jitter is an issue that makes it unusable for smooth motion data source.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

Im surprised this feature request has been marked as stale. Landmark jittering is a major issue. Although the overall tracking capability is impressive, It makes mediapipe unusable in my opinion. The SMOOTH_LANDMARKS feature is available in solutions.holistic. Why is it not available in solutions.hands?

Closing as stale. Please reopen if you'd like to work on this further.

I've begun investigating implementing one euro temporal filtering but it introduces lag issues.

@willanie have you been able to implement this filtering? I'm also trying to fix the jitter with filtering, however, at the moment I only have an implementation with FFTs that works on a recorded clip, i.e. as a form of post-processing. As a next step I want to try to apply the FFT with a sliding window (or something of this sort)

@willanie @koegl did you have any success with jitter reduce? I use one-euro-filter that helps a little, but Mediapipe hands is still unusable to me at this stage because of jitter.

one-euro-filter

what is that one-euro-filter?

@willanie,

This is part of Face Landmarker Tasks API. Please go through the our new solutions and let us know if you encounter the similar jitter behaviour.

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

This issue was closed due to lack of activity after being marked stale for past 7 days.