MediaPipeUnityPlugin icon indicating copy to clipboard operation
MediaPipeUnityPlugin copied to clipboard

TongueOut is missing in blendshapes output

Open Hakazaba opened this issue 2 years ago • 4 comments

Description

Hey, im just exploring the amazing new face landmarker blendshape and matrices and i found what seems to be a little bug, the package im working from is a little bit old (#146 on linux) so it might have been fixed, but just encase it isn't i thought it worth mentioning.

The blendshape array lacks 52 - TongueOut. Likely a buffer overflow or something with the array being set to 52 instead of 53

Hakazaba avatar Oct 23 '23 12:10 Hakazaba

Could you let me know which part of the code you're referring to?

homuler avatar Oct 23 '23 13:10 homuler

When accessing FaceLandmarkerResult.BlendShapes, it only has 52 indexes, it should have 53.

It may seem like it should be 52 according to this: https://storage.googleapis.com/mediapipe-assets/Model%20Card%20Blendshape%20V2.pdf however 0 is actually 'neutral' and not listed there. You can see this by printing the collection names of each collection in that list

Hakazaba avatar Oct 25 '23 10:10 Hakazaba

Same as https://github.com/google/mediapipe/issues/4403

By the way, what are you talking about when you mention 'buffer overflow'? Vulnerability?

homuler avatar Oct 25 '23 14:10 homuler

Never mind, i had thought the model would be outputting 53 blendshapes, then inserted into an array with a length of 52, thereby causing a buffer overflow and a loss of the final blendshape, due to the documentation saying there are only 52 blendshapes without including the 'neutral' blendshape at 0. It seems the problem is more endemic than that.

Hakazaba avatar Oct 26 '23 00:10 Hakazaba