mediapipe icon indicating copy to clipboard operation
mediapipe copied to clipboard

Published 20 hours ago • verified publisher icon google-ai-edge

image_format=<ImageFormat.SRGB: 1> __init__(): incompatible constructor arguments.

Open aerinkim opened this issue 1 year ago • 1 comments

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

None

OS Platform and Distribution

Ubuntu 20.04.4 LTS

MediaPipe Tasks SDK version

mediapipe-0.10.11

Task name (e.g. Image classification, Gesture recognition etc.)

tasks.vision.FaceLandmarker

Programming Language and version (e.g. C++, Python, Java)

Python 3.8 (Conda)

Describe the actual behavior

Exception has occurred: TypeError init(): incompatible constructor arguments. The following argument types are supported: 1. mediapipe.python._framework_bindings.image.Image(image_format: mediapipe::ImageFormat_Format, data: numpy.ndarray[numpy.uint8]) 2. mediapipe.python._framework_bindings.image.Image(image_format: mediapipe::ImageFormat_Format, data: numpy.ndarray[numpy.uint16]) 3. mediapipe.python._framework_bindings.image.Image(image_format: mediapipe::ImageFormat_Format, data: numpy.ndarray[numpy.float32]) Invoked with: kwargs: image_format=<ImageFormat.SRGB: 1>,

Describe the expected behaviour

should not get exception.

Standalone code/steps you may have used to try to get what you need

import mediapipe as mp
from mediapipe.tasks import python
from mediapipe.tasks.python import vision

BaseOptions = mp.tasks.BaseOptions
FaceLandmarker = mp.tasks.vision.FaceLandmarker
FaceLandmarkerOptions = mp.tasks.vision.FaceLandmarkerOptions

base_options = python.BaseOptions(model_asset_path="/home/ubuntu/Projects/miraflow-ml-aerin/SyncNet288Experiment/weights/face_landmarker_v2_with_blendshapes.task")
options = FaceLandmarkerOptions(base_options=base_options,
                                      output_face_blendshapes=True,
                                      output_facial_transformation_matrixes=True,
                                      num_faces=1)
face_landmarks_detector = FaceLandmarker.create_from_options(options)


def get_mask_from_lower_face(image):
    """
    Generate and save a mask that covers the lower face, including lips and surrounding areas but excluding the nose.

    Args:
        image: numpy array of an image (H,W,C)

    Returns:
        A uint8 numpy array with the same height and width of the input image,
        containing a binary mask of the lower face
    """
    image = np.asarray(image, dtype=np.float32)

    # Initialize mask
    mask = np.zeros((image.shape[0], image.shape[1]), dtype=np.uint8)

    # detect face landmarks    mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=image)
    detection = face_landmarks_detector.detect(mp_image)

I tried the image with numpy array - uint8, unit16, float32 but I keep getting this error.



### Other info / Complete Logs

```shell
Exception has occurred: TypeError
__init__(): incompatible constructor arguments. The following argument types are supported:
    1. mediapipe.python._framework_bindings.image.Image(image_format: mediapipe::ImageFormat_Format, data: numpy.ndarray[numpy.uint8])
    2. mediapipe.python._framework_bindings.image.Image(image_format: mediapipe::ImageFormat_Format, data: numpy.ndarray[numpy.uint16])
    3. mediapipe.python._framework_bindings.image.Image(image_format: mediapipe::ImageFormat_Format, data: numpy.ndarray[numpy.float32])

Invoked with: kwargs: image_format=<ImageFormat.SRGB: 1>, data=array([[[0.84313726, 0.9137255 , 0.9098039 ],
        [0.84313726, 0.9137255 , 0.9098039 ],
        [0.84705883, 0.91764706, 0.9137255 ],
        ...,
        [0.88235295, 0.95686275, 0.9764706 ],
        [0.88235295, 0.95686275, 0.9764706 ],
        [0.8862745 , 0.9647059 , 0.9843137 ]],

       [[0.8117647 , 0.88235295, 0.8862745 ],
        [0.8117647 , 0.8862745 , 0.8862745 ],
        [0.8156863 , 0.8901961 , 0.8901961 ],
        ...,
        [0.8627451 , 0.9411765 , 0.9607843 ],
        [0.8627451 , 0.9411765 , 0.9607843 ],
        [0.8627451 , 0.9411765 , 0.9607843 ]],

       [[0.8156863 , 0.8862745 , 0.8901961 ],
        [0.8156863 , 0.8862745 , 0.8901961 ],
        [0.8117647 , 0.88235295, 0.8862745 ],
        ...,
        [0.85490197, 0.93333334, 0.9529412 ],
        [0.85490197, 0.93333334, 0.9529412 ],
        [0.85882354, 0.9372549 , 0.95686275]],

       ...,

       [[0.8117647 , 0.88235295, 0.8862745 ],
        [0.8117647 , 0.88235295, 0.8862745 ],
        [0.8117647 , 0.88235295, 0.8862745 ],
        ...,
        [0.3254902 , 0.8862745 , 0.91764706],
        [0.3254902 , 0.8862745 , 0.91764706],
        [0.32156864, 0.8784314 , 0.90588236]],

       [[0.80784315, 0.8784314 , 0.88235295],
        [0.80784315, 0.8784314 , 0.88235295],
        [0.8117647 , 0.88235295, 0.8862745 ],
        ...,
        [0.3254902 , 0.8862745 , 0.91764706],
        [0.3254902 , 0.8862745 , 0.91764706],
        [0.32156864, 0.8784314 , 0.90588236]],

       [[0.8117647 , 0.88235295, 0.8862745 ],
        [0.8117647 , 0.88235295, 0.8862745 ],
        [0.8156863 , 0.8862745 , 0.8901961 ],
        ...,
        [0.31764707, 0.8784314 , 0.90588236],
        [0.31764707, 0.8784314 , 0.90588236],
        [0.32941177, 0.8862745 , 0.9137255 ]]], dtype=float32)
  File "/home/ubuntu/Projects/miraflow-ml-aerin/SyncNet288Experiment/utils_aerin.py", line 332, in get_mask_from_lower_face
    mp_image = mp.Image(image_format=mp.ImageFormat.SRGB, data=image)
  File "/home/ubuntu/Projects/miraflow-ml-aerin/SyncNet288Experiment/hq_train_speed_up_mask.py", line 269, in apply_mouth_mask
    mask = get_mask_from_lower_face(image)
  File "/home/ubuntu/Projects/miraflow-ml-aerin/SyncNet288Experiment/hq_train_speed_up_mask.py", line 320, in train
    x = apply_mouth_mask(x) #x:[B, 3+3, T, H, W]
  File "/home/ubuntu/Projects/miraflow-ml-aerin/SyncNet288Experiment/hq_train_speed_up_mask.py", line 596, in <module>
    train(device, model, disc, train_data_loader, test_data_loader, optimizer, disc_optimizer,
TypeError: __init__(): incompatible constructor arguments. The following argument types are supported:
    1. mediapipe.python._framework_bindings.image.Image(image_format: mediapipe::ImageFormat_Format, data: numpy.ndarray[numpy.uint8])
    2. mediapipe.python._framework_bindings.image.Image(image_format: mediapipe::ImageFormat_Format, data: numpy.ndarray[numpy.uint16])
    3. mediapipe.python._framework_bindings.image.Image(image_format: mediapipe::ImageFormat_Format, data: numpy.ndarray[numpy.float32])

Invoked with: kwargs: image_format=<ImageFormat.SRGB: 1>, data=array([[[0.84313726, 0.9137255 , 0.9098039 ],
        [0.84313726, 0.9137255 , 0.9098039 ],
        [0.84705883, 0.91764706, 0.9137255 ],
        ...,
        [0.88235295, 0.95686275, 0.9764706 ],
        [0.88235295, 0.95686275, 0.9764706 ],
        [0.8862745 , 0.9647059 , 0.9843137 ]],

       [[0.8117647 , 0.88235295, 0.8862745 ],
        [0.8117647 , 0.8862745 , 0.8862745 ],
        [0.8156863 , 0.8901961 , 0.8901961 ],
        ...,
        [0.8627451 , 0.9411765 , 0.9607843 ],
        [0.8627451 , 0.9411765 , 0.9607843 ],
        [0.8627451 , 0.9411765 , 0.9607843 ]],

       [[0.8156863 , 0.8862745 , 0.8901961 ],
        [0.8156863 , 0.8862745 , 0.8901961 ],
        [0.8117647 , 0.88235295, 0.8862745 ],
        ...,
        [0.85490197, 0.93333334, 0.9529412 ],
        [0.85490197, 0.93333334, 0.9529412 ],
        [0.85882354, 0.9372549 , 0.95686275]],

       ...,

       [[0.8117647 , 0.88235295, 0.8862745 ],
        [0.8117647 , 0.88235295, 0.8862745 ],
        [0.8117647 , 0.88235295, 0.8862745 ],
        ...,
        [0.3254902 , 0.8862745 , 0.91764706],
        [0.3254902 , 0.8862745 , 0.91764706],
        [0.32156864, 0.8784314 , 0.90588236]],

       [[0.80784315, 0.8784314 , 0.88235295],
        [0.80784315, 0.8784314 , 0.88235295],
        [0.8117647 , 0.88235295, 0.8862745 ],
        ...,
        [0.3254902 , 0.8862745 , 0.91764706],
        [0.3254902 , 0.8862745 , 0.91764706],
        [0.32156864, 0.8784314 , 0.90588236]],

       [[0.8117647 , 0.88235295, 0.8862745 ],
        [0.8117647 , 0.88235295, 0.8862745 ],
        [0.8156863 , 0.8862745 , 0.8901961 ],
        ...,
        [0.31764707, 0.8784314 , 0.90588236],
        [0.31764707, 0.8784314 , 0.90588236],
        [0.32941177, 0.8862745 , 0.9137255 ]]], dtype=float32)

aerinkim avatar May 06 '24 04:05 aerinkim

Hi @aerinkim,

We are aware of this behavior. Prior to creating an mp.Image, you should preprocess the frame by converting something like it to a numpy array using frame = np.array(frame). We have already confirmed with the team that there are no plans to extend support to numpy views.

For, more understanding you may refer to this issue #5251.

Thank you!!

kuaashish avatar May 08 '24 08:05 kuaashish

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

github-actions[bot] avatar May 16 '24 01:05 github-actions[bot]

This issue was closed due to lack of activity after being marked stale for past 7 days.

github-actions[bot] avatar May 24 '24 01:05 github-actions[bot]

Are you satisfied with the resolution of your issue? Yes No

google-ml-butler[bot] avatar May 24 '24 01:05 google-ml-butler[bot]