tongueOut is missing in blendshapes output

[jiangwei221]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

No

OS Platform and Distribution

M2 Mac

MediaPipe Tasks SDK version

No response

Task name (e.g. Image classification, Gesture recognition etc.)

Face Landmarks

Programming Language and version (e.g. C++, Python, Java)

Python

Describe the actual behavior

I tried to run the sample code at: https://github.com/googlesamples/mediapipe/blob/main/examples/face_landmarker/python/%5BMediaPipe_Python_Tasks%5D_Face_Landmarker.ipynb
The `tongueOut` is missing in the blendshapes outputs, and the blend shape at index 0 is `_neutral `. Could you add more details about `_neutral `?

Describe the expected behaviour

`tongueOut` should be in the blendshapes outputs, according to the tech spec at: https://storage.googleapis.com/mediapipe-assets/Model%20Card%20Blendshape%20V2.pdf

Standalone code/steps you may have used to try to get what you need

https://github.com/googlesamples/mediapipe/blob/main/examples/face_landmarker/python/%5BMediaPipe_Python_Tasks%5D_Face_Landmarker.ipynb

Other info / Complete Logs

No response

[TigerHix]

Reproduced on the JavaScript runtime too. Here are the FaceLandmarkerResult landmarks:

{
    "categories": [
        {
            "index": 0,
            "score": 4.6296000277834537e-7,
            "categoryName": "_neutral",
            "displayName": ""
        },
        {
            "index": 1,
            "score": 0.00009300197416450828,
            "categoryName": "browDownLeft",
            "displayName": ""
        },
        {
            "index": 2,
            "score": 0.00015640698256902397,
            "categoryName": "browDownRight",
            "displayName": ""
        },
        {
            "index": 3,
            "score": 0.6249369382858276,
            "categoryName": "browInnerUp",
            "displayName": ""
        },
        {
            "index": 4,
            "score": 0.7325420379638672,
            "categoryName": "browOuterUpLeft",
            "displayName": ""
        },
        {
            "index": 5,
            "score": 0.588916003704071,
            "categoryName": "browOuterUpRight",
            "displayName": ""
        },
        {
            "index": 6,
            "score": 0.000046172110160114244,
            "categoryName": "cheekPuff",
            "displayName": ""
        },
        {
            "index": 7,
            "score": 1.7091801396418305e-7,
            "categoryName": "cheekSquintLeft",
            "displayName": ""
        },
        {
            "index": 8,
            "score": 2.8026576615047816e-7,
            "categoryName": "cheekSquintRight",
            "displayName": ""
        },
        {
            "index": 9,
            "score": 0.08379079401493073,
            "categoryName": "eyeBlinkLeft",
            "displayName": ""
        },
        {
            "index": 10,
            "score": 0.09008345007896423,
            "categoryName": "eyeBlinkRight",
            "displayName": ""
        },
        {
            "index": 11,
            "score": 0.04183768108487129,
            "categoryName": "eyeLookDownLeft",
            "displayName": ""
        },
        {
            "index": 12,
            "score": 0.04601563885807991,
            "categoryName": "eyeLookDownRight",
            "displayName": ""
        },
        {
            "index": 13,
            "score": 0.023890702053904533,
            "categoryName": "eyeLookInLeft",
            "displayName": ""
        },
        {
            "index": 14,
            "score": 0.1881195306777954,
            "categoryName": "eyeLookInRight",
            "displayName": ""
        },
        {
            "index": 15,
            "score": 0.20005182921886444,
            "categoryName": "eyeLookOutLeft",
            "displayName": ""
        },
        {
            "index": 16,
            "score": 0.01575462333858013,
            "categoryName": "eyeLookOutRight",
            "displayName": ""
        },
        {
            "index": 17,
            "score": 0.1794263869524002,
            "categoryName": "eyeLookUpLeft",
            "displayName": ""
        },
        {
            "index": 18,
            "score": 0.20379243791103363,
            "categoryName": "eyeLookUpRight",
            "displayName": ""
        },
        {
            "index": 19,
            "score": 0.509730339050293,
            "categoryName": "eyeSquintLeft",
            "displayName": ""
        },
        {
            "index": 20,
            "score": 0.45855653285980225,
            "categoryName": "eyeSquintRight",
            "displayName": ""
        },
        {
            "index": 21,
            "score": 0.008235435001552105,
            "categoryName": "eyeWideLeft",
            "displayName": ""
        },
        {
            "index": 22,
            "score": 0.006482802797108889,
            "categoryName": "eyeWideRight",
            "displayName": ""
        },
        {
            "index": 23,
            "score": 0.0002498253888916224,
            "categoryName": "jawForward",
            "displayName": ""
        },
        {
            "index": 24,
            "score": 0.0012172190472483635,
            "categoryName": "jawLeft",
            "displayName": ""
        },
        {
            "index": 25,
            "score": 0.06601714342832565,
            "categoryName": "jawOpen",
            "displayName": ""
        },
        {
            "index": 26,
            "score": 0.0005452344776131213,
            "categoryName": "jawRight",
            "displayName": ""
        },
        {
            "index": 27,
            "score": 0.006105016451328993,
            "categoryName": "mouthClose",
            "displayName": ""
        },
        {
            "index": 28,
            "score": 0.03788747265934944,
            "categoryName": "mouthDimpleLeft",
            "displayName": ""
        },
        {
            "index": 29,
            "score": 0.008929312229156494,
            "categoryName": "mouthDimpleRight",
            "displayName": ""
        },
        {
            "index": 30,
            "score": 0.0000010028247743321117,
            "categoryName": "mouthFrownLeft",
            "displayName": ""
        },
        {
            "index": 31,
            "score": 0.0000018083688928527408,
            "categoryName": "mouthFrownRight",
            "displayName": ""
        },
        {
            "index": 32,
            "score": 0.011962022632360458,
            "categoryName": "mouthFunnel",
            "displayName": ""
        },
        {
            "index": 33,
            "score": 0.006822296418249607,
            "categoryName": "mouthLeft",
            "displayName": ""
        },
        {
            "index": 34,
            "score": 0.014128030277788639,
            "categoryName": "mouthLowerDownLeft",
            "displayName": ""
        },
        {
            "index": 35,
            "score": 0.0300290584564209,
            "categoryName": "mouthLowerDownRight",
            "displayName": ""
        },
        {
            "index": 36,
            "score": 0.025603007525205612,
            "categoryName": "mouthPressLeft",
            "displayName": ""
        },
        {
            "index": 37,
            "score": 0.019609909504652023,
            "categoryName": "mouthPressRight",
            "displayName": ""
        },
        {
            "index": 38,
            "score": 0.15044556558132172,
            "categoryName": "mouthPucker",
            "displayName": ""
        },
        {
            "index": 39,
            "score": 0.0007236231467686594,
            "categoryName": "mouthRight",
            "displayName": ""
        },
        {
            "index": 40,
            "score": 0.008093741722404957,
            "categoryName": "mouthRollLower",
            "displayName": ""
        },
        {
            "index": 41,
            "score": 0.02538474276661873,
            "categoryName": "mouthRollUpper",
            "displayName": ""
        },
        {
            "index": 42,
            "score": 0.0005738880718126893,
            "categoryName": "mouthShrugLower",
            "displayName": ""
        },
        {
            "index": 43,
            "score": 0.0018722142558544874,
            "categoryName": "mouthShrugUpper",
            "displayName": ""
        },
        {
            "index": 44,
            "score": 0.00007578419172205031,
            "categoryName": "mouthSmileLeft",
            "displayName": ""
        },
        {
            "index": 45,
            "score": 0.00008054033241933212,
            "categoryName": "mouthSmileRight",
            "displayName": ""
        },
        {
            "index": 46,
            "score": 0.000020466661226237193,
            "categoryName": "mouthStretchLeft",
            "displayName": ""
        },
        {
            "index": 47,
            "score": 0.0007130264420993626,
            "categoryName": "mouthStretchRight",
            "displayName": ""
        },
        {
            "index": 48,
            "score": 0.00019434058049228042,
            "categoryName": "mouthUpperUpLeft",
            "displayName": ""
        },
        {
            "index": 49,
            "score": 0.00018397865642327815,
            "categoryName": "mouthUpperUpRight",
            "displayName": ""
        },
        {
            "index": 50,
            "score": 0.0000022188494313013507,
            "categoryName": "noseSneerLeft",
            "displayName": ""
        },
        {
            "index": 51,
            "score": 1.455517519843852e-7,
            "categoryName": "noseSneerRight",
            "displayName": ""
        }
    ],
    "headIndex": -1,
    "headName": ""
}

[jays0606]

From the 478 facial landmarks, tongue is not included in the mesh. They probably intentionally skipped tongueOut.

[TigerHix]

From the 478 facial landmarks, tongue is not included in the mesh. They probably intentionally skipped tongueOut.

The model documentation here did say the blendshape is available albeit predicted by the FaceMesh model.

[ayushgdev]

Hello @jiangwei221 We are able to reproduce this issue at our end. Please allow us sometime to get back to you after discussing internally.

[zihaomu]

Any update on this issue?

[c4b4d4]

Is there an interest on adding any soon the tongueOut blendshape?

[beenotung]

It would be a great value add to support on the keypoints and blendshapes of tongue. What is needed to proceed? If we need some annotated data, we can crowd source it :muscle: