How about only using keypoints and no masking?

[stephanschulz]

In your current code:

• you define a rectangle on the forehead.
• Then use the 4 corners and do openCV optical flow keypoint tracking to get smoother tracking
• use estimateAffine2D to rectify the tracked area.
• Based on this rectangle you create a mask and analyze only the RGB image based on the mask.

What do you think how accurate the PPG analysis would be if i defined 1000 optical flow keypoints and got the mean RGB value from those 1000 pixel points? The keypoint tracking ensures that we are reading the same skin pixels throughout consecutive frame. But i don't know if 1000 points is too low or if we need a continuous 2D space.

Thanks for your advice.

https://user-images.githubusercontent.com/1054816/109538269-e243be80-7a8d-11eb-8e48-8514a1af3343.mov

[prouast]

This would certainly be worth exploring and I'd be interested to hear whether it works.

What I can add:

• The reason to use a continuous area of skin pixels is that the average cancels out random fluctuations in individual pixels
• I am using the rectangle on the forehead because the signal is present, and it is straightforward to hardcode based on the face detection coordinates