mediapipe

mediapipe copied to clipboard

I'm working with mediapipe face mesh landmarks model. What I want is to find the 468 landmarks for a face and then filter out any faces with occluded landmarks. The model has these attributes defined as landmarks 'visibility' and 'presence'. But when I print out these values for all the landmarks they appear to be 0. For a frontal face they should atleast have a value greater than 0.5.

Hi @waleedrazakhan92 , could you please share us in which platform you are trying this solution.

Hi @waleedrazakhan92 , could you please share us in which platform you are trying this solution.

@sureshdagooglecom I'm using google colab.

Hi @waleedrazakhan92 , As model predicts presence and visibility as x,y values ,to get propobalities you need to apply the sigmoid function on these values so you will get values in the range [0, 1]. We have to do this in the subsequent calculators just after the model inference in the graph.

So reasonable threshold will be 0.5 if you apply the sigmoid.

Hi @waleedrazakhan92 , As model predicts presence and visibility as x,y values ,to get propobalities you need to apply the sigmoid function on these values so you will get values in the range [0, 1]. We have to do this in the subsequent calculators just after the model inference in the graph.

So reasonable threshold will be 0.5 if you apply the sigmoid.

@sureshdagooglecom can you guide me as to which calculator file do I need to change and where do i put the sigmoid function.? I'm looking at the calculators folder (mediapipe>calculators) but i'm not sure which file to edit and add sigmoid. I'm working in colab with python environment.

Also can you clear this up that whether the 'visibility' and 'presence' attributes are available for face mesh model or not?

Also the attribute landmark.HasField('visibility') and landmark.HasField('presence') are also false for all the landmarks.

@sureshdagooglecom - Hi Suresh, We are building a Python script to filter some face images and also wanted to use the Landmarks as a method.

We assumed there existed a "per landmark" method or argument which; IF a landmark was not able to be applied/used or it was somehow blocked VS predicted and placed.

Usecase: We have some images were part of the Chin or Mouth is missing/covered and we want to filter that from entering a dataset.

Can you point me in the direction of sample code or link to help me (Python)

Hi @jimb2834 ,can you share me your code changes to investigate further on this issue.

Hi @waleedrazakhan92 , As model predicts presence and visibility as x,y values ,to get propobalities you need to apply the sigmoid function on these values so you will get values in the range [0, 1]. We have to do this in the subsequent calculators just after the model inference in the graph.

So reasonable threshold will be 0.5 if you apply the sigmoid.

Hi @sureshdagooglecom please guide me through the process of adding the calculators in the code. I believe the inference is importing the FaceMesh class from this file(https://github.com/google/mediapipe/blob/master/mediapipe/python/solutions/face_mesh.py) and in the imports section there are a number of 'calculators' imported from 'mediapipe.calculators' that calculate the final landmarks on the face.

What i don't understand is, to utilize the 'visibility' and 'presence' attribute of the face mesh model(assuming the model does support the visibility and presence attributes), which calculator do i need to import from 'mediapipe.calculators'?

I also believe that we need to import the calculator in 'https://github.com/google/mediapipe/blob/master/mediapipe/python/solution_base.py' and add it to : CALCULATOR_TO_OPTIONS = { 'ConstantSidePacketCalculator': constant_side_packet_calculator_pb2.ConstantSidePacketCalculatorOptions, 'ImageTransformationCalculator': image_transformation_calculator_pb2 .ImageTransformationCalculatorOptions, 'LandmarksSmoothingCalculator': landmarks_smoothing_calculator_pb2.LandmarksSmoothingCalculatorOptions, 'LogicCalculator': logic_calculator_pb2.LogicCalculatorOptions, 'ThresholdingCalculator': thresholding_calculator_pb2.ThresholdingCalculatorOptions, 'TensorsToDetectionsCalculator': tensors_to_detections_calculator_pb2 .TensorsToDetectionsCalculatorOptions, 'Lift2DFrameAnnotationTo3DCalculator': lift_2d_frame_annotation_to_3d_calculator_pb2 .Lift2DFrameAnnotationTo3DCalculatorOptions, } No which calculator do i need to pass here and what other changes do i need to make to make the visibility and the presence calculators available for my face mesh model inference?

Please any help and example would be appreciated!

@sureshdagooglecom - Thank you for your help - I will ask my team lead if we can. But I was following this guide here -

https://towardsdatascience.com/face-landmarks-detection-with-mediapipe-facemesh-555fa2e10b06

We are doing the same thing as @waleedrazakhan92

Our use case is to pass an image which may have part of the "jaw line" obfuscated, or maybe one eye is covered etc.. And use the landmark "visibility" or "presence" to indicate this. I thought we could use the landmarks to tell us this info and then I can filter out these bad images.

Qn: Is this possible to know if a landmark was predicted or actually found? i.e. present or visible ?

"we will invoke the model.estimateFaces() function to get the predicted face landmark positions. The estimateFaces() " - So its not "actually" found landmarks ? And is there a method to actually state if a landmark was "present" or "predicted"

We found this snippet and wanted to know if for python there is equivalent function:

def get_landmark_coordinates_bypixel(image, landmark_list: landmark_pb2.NormalizedLandmarkList, mark_index: int):
    if not landmark_list:
        return
    if image.shape[2] != RGB_CHANNELS:
        raise ValueError('Input image must contain three channel rgb data.')
    image_rows, image_cols, _ = image.shape
    idx_to_coordinates = {}
    landmark = landmark_list.landmark[mark_index]
    if ((landmark.HasField('visibility') and
         landmark.visibility < VISIBILITY_THRESHOLD) or
            (landmark.HasField('presence') and
             landmark.presence < PRESENCE_THRESHOLD)):
        return
    landmark_px = _normalized_to_pixel_coordinates(landmark.x, landmark.y,
                                                   image_cols, image_rows)
    return landmark_px

@cdibona @sureshdagooglecom - Is it true that in python you can not get the landmark visibility or presence values?

Hi @waleedrazakhan92 , As model predicts presence and visibility as x,y values ,to get propobalities you need to apply the sigmoid function on these values so you will get values in the range [0, 1]. We have to do this in the subsequent calculators just after the model inference in the graph.

So reasonable threshold will be 0.5 if you apply the sigmoid.

EDIT: I'm installing the medipipe using "pip install mediapipe" and then editing the source files in the package itself.

Hi @sureshdagooglecom, You said we need to add the sigmoid function to the calculators to get the visibility and presence values between 0 and 1. So i believe that would be the 'tensors_to_landmarks_calculator_pb2.py' calculator. Upon investigating the code I believe that its already applying the sigmoid function on the values. If you see the lines 143-147 isn't it already applying the sigmoid function to the values?

Also in the 'solution_base.py' file I've imported the tensors_to_landmarks_calculator as follows:

After making these changes there is still no change in the output and the visibility and presence values are still not present. Please advise me on what else do i need to change.

Here is the link to the code notebook. https://colab.research.google.com/drive/1mLYRm5NJbWACczzl_syAetYiypLlOkeV?usp=sharing

@sureshdagooglecom - Any word if this is working for anyone?

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

@waleedrazakhan92 - have you solved this ? The topic is stale ? What did the Mediapipe team in Slack say ?

@sureshdagooglecom - Hello, Could you please let me know if you sell support tickets to review and fix the script ?

@waleedrazakhan92 - have you solved this ? The topic is stale ? What did the Mediapipe team in Slack say ?

@jimb2834 I haven't received a solution yet. There are however a number of similar issues but no helpful solutions have been presented by the team. Also there is no clear cut indication whether the face mesh model indeed outputs 'visibility' and 'presence' values or not. I also did try editing the graph file as mentioned in #3200 but the graph doesn't build. I'm confused why the mediapipe team hasn't been helpful in all this since the issue seems to be recurring.

bump to stay open and get a reply

@sureshdagooglecom - Hi Suresh, Would you be willing to re-assign this to someone with more availability ?

Hi, I also have the very same problem, I would love to find a way to have a "visibility", "presence", or a "confidence score" for each individual landmark, I have been trying without any success, did anyone manage to find a solution?

@zeina-abuaisheh - We as you can see have been waiting >5 months now and no replys. We have no known solution yet the API docs still state Visibility in the facemesh lib docs

@waleedrazakhan92 - Any update about this visibility and presence for each landmark issue? We are still facing this issue.

@waleedrazakhan92 - Any update about this visibility and presence for each landmark issue? We are still facing this issue.

@tsnagtode21 No its not resolved yet.

no update from my side, if anyone found a solution, that would be nice to share it with us

same here, solution would be much appreciated

Hello @waleedrazakhan92, We are upgrading the MediaPipe Legacy Solutions to new MediaPipe solutions However, the libraries, documentation, and source code for all the MediapPipe Legacy Solutions will continue to be available in our GitHub repository and through library distribution services, such as Maven and NPM.

You can continue to use those legacy solutions in your applications if you choose. Though, we would request you to check new MediaPipe solutions which can help you more easily build and customize ML solutions for your applications. These new solutions will provide a superset of capabilities available in the legacy solutions. Thank you

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

This issue was closed due to lack of activity after being marked stale for past 7 days.

Are you satisfied with the resolution of your issue? Yes No

Hello @waleedrazakhan92, We are upgrading the MediaPipe Legacy Solutions to new MediaPipe solutions However, the libraries, documentation, and source code for all the MediapPipe Legacy Solutions will continue to be available in our GitHub repository and through library distribution services, such as Maven and NPM.

You can continue to use those legacy solutions in your applications if you choose. Though, we would request you to check new MediaPipe solutions which can help you more easily build and customize ML solutions for your applications. These new solutions will provide a superset of capabilities available in the legacy solutions. Thank you

@kuaashish so the conclusion is the newer model has no way to determine confidence/visibility/presence of keypoints? I'm unclear if you're suggesting as a conclusion to this issue that the Legacy version does have this functionality?

I have the same problem. I want to check if the face is occluded or not. I print the landmark.visibility and landmark.presence and they always are 0 whatever face's occluded or normal. I would like to know what do they for ?

facing same issue is medaipipe team doing something about it