mediapipe

mediapipe copied to clipboard

Please make sure that this is a solution issue.

System information (Please provide as much relevant information as possible)

• OS Platform and Distribution linux Ubuntu 21.04
• MediaPipe version: v0.8.10.2
• Bazel version: v1.12.0
• Solution (e.g. FaceMesh, Pose, Holistic): Face Effect (Face Mesh)
• Programming Language and version: C++ 10.3.0

Describe the expected behavior: I'm trying to run face effect demo to fit mask or eye glasses on my face.

Standalone code you may have used to try to get what you need : I have built graph by command below:

bazel build -c opt --copt -DMESA_EGL_NO_X11_HEADERS --copt -DEGL_NO_X11 mediapipe/examples/desktop/_face_effect:face_effect_gpu --verbose_failures

And trying to run the graph with command below:

GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/_face_effect/face_effect_gpu --calculator_graph_config_file=mediapipe/graphs/face_effect/face_effect_gpu.pbtxt

However it fails to start. Log is at the end of the post.

Other info / Complete Logs : My BUILD file is as below:

licenses(["notice"])

package(default_visibility = ["//mediapipe/examples:__subpackages__"])

cc_binary(
    name = "face_effect_gpu",
    deps = [
        "//mediapipe/examples/desktop:demo_run_graph_main_gpu",
        "//mediapipe/graphs/face_effect:face_effect_gpu_deps",
    ],
)

Finally log is below:

I20220725 20:54:06.296331 358140 demo_run_graph_main_gpu.cc:52] Get calculator graph config contents: # MediaPipe graph that applies a face effect to the input video stream.

# GPU buffer. (GpuBuffer)
input_stream: "input_video"

# An integer, which indicate which effect is selected. (int)
#
# If `selected_effect_id` is `0`, the Axis effect is selected.
# If `selected_effect_id` is `1`, the Facepaint effect is selected.
# If `selected_effect_id` is `2`, the Glasses effect is selected.
#
# No other values are allowed for `selected_effect_id`.
input_stream: "selected_effect_id"

# Indicates whether to use the face detection as the input source. (bool)
#
# If `true`, the face detection pipeline will be used to produce landmarks.
# If `false`, the face landmark pipeline will be used to produce landmarks.
input_side_packet: "use_face_detection_input_source"

# Output image with rendered results. (GpuBuffer)
output_stream: "output_video"

# A list of geometry data for a single detected face.
#
# NOTE: there will not be an output packet in this stream for this particular
# timestamp if none of faces detected.
#
# (std::vector<face_geometry::FaceGeometry>)
output_stream: "multi_face_geometry"

# Throttles the images flowing downstream for flow control. It passes through
# the very first incoming image unaltered, and waits for downstream nodes
# (calculators and subgraphs) in the graph to finish their tasks before it
# passes through another image. All images that come in while waiting are
# dropped, limiting the number of in-flight images in most part of the graph to
# 1. This prevents the downstream nodes from queuing up incoming images and data
# excessively, which leads to increased latency and memory usage, unwanted in
# real-time mobile applications. It also eliminates unnecessarily computation,
# e.g., the output produced by a node may get dropped downstream if the
# subsequent nodes are still busy processing previous inputs.
node {
  calculator: "FlowLimiterCalculator"
  input_stream: "input_video"
  input_stream: "FINISHED:output_video"
  input_stream_info: {
    tag_index: "FINISHED"
    back_edge: true
  }
  output_stream: "throttled_input_video"
}

# Generates an environment that describes the current virtual scene.
node {
  calculator: "FaceGeometryEnvGeneratorCalculator"
  output_side_packet: "ENVIRONMENT:environment"
  node_options: {
    [type.googleapis.com/mediapipe.FaceGeometryEnvGeneratorCalculatorOptions] {
      environment: {
        origin_point_location: TOP_LEFT_CORNER
        perspective_camera: {
          vertical_fov_degrees: 63.0  # 63 degrees
          near: 1.0  # 1cm
          far: 10000.0  # 100m
        }
      }
    }
  }
}

# Computes the face geometry for a single face. The input source is defined
# through `use_face_detection_input_source`.
node {
  calculator: "SwitchContainer"
  input_stream: "IMAGE:throttled_input_video"
  input_side_packet: "ENABLE:use_face_detection_input_source"
  input_side_packet: "ENVIRONMENT:environment"
  output_stream: "MULTI_FACE_GEOMETRY:multi_face_geometry"
  node_options: {
    [type.googleapis.com/mediapipe.SwitchContainerOptions] {
      contained_node: {
        calculator: "SingleFaceGeometryFromLandmarksGpu"
      }
      contained_node: {
        calculator: "SingleFaceGeometryFromDetectionGpu"
      }
    }
  }
}

# Renders the selected effect based on `selected_effect_id`.
node {
  calculator: "SwitchContainer"
  input_stream: "SELECT:selected_effect_id"
  input_stream: "IMAGE_GPU:throttled_input_video"
  input_stream: "MULTI_FACE_GEOMETRY:multi_face_geometry"
  input_side_packet: "ENVIRONMENT:environment"
  output_stream: "IMAGE_GPU:output_video"
  node_options: {
    [type.googleapis.com/mediapipe.SwitchContainerOptions] {
      contained_node: {
        calculator: "FaceGeometryEffectRendererCalculator"
        node_options: {
          [type.googleapis.com/mediapipe.FaceGeometryEffectRendererCalculatorOptions] {
            effect_texture_path: "mediapipe/graphs/face_effect/data/axis.pngblob"
            effect_mesh_3d_path: "mediapipe/graphs/face_effect/data/axis.binarypb"
          }
        }
      }
      contained_node: {
        calculator: "FaceGeometryEffectRendererCalculator"
        node_options: {
          [type.googleapis.com/mediapipe.FaceGeometryEffectRendererCalculatorOptions] {
            effect_texture_path: "mediapipe/graphs/face_effect/data/facepaint.pngblob"
          }
        }
      }
      contained_node: {
        calculator: "FaceGeometryEffectRendererCalculator"
        node_options: {
          [type.googleapis.com/mediapipe.FaceGeometryEffectRendererCalculatorOptions] {
            effect_texture_path: "mediapipe/graphs/face_effect/data/glasses.pngblob"
            effect_mesh_3d_path: "mediapipe/graphs/face_effect/data/glasses.binarypb"
          }
        }
      }
    }
  }
}

I20220725 20:54:06.297303 358140 demo_run_graph_main_gpu.cc:58] Initialize the calculator graph.
I20220725 20:54:06.304211 358140 demo_run_graph_main_gpu.cc:62] Initialize the GPU.
I20220725 20:54:06.317529 358140 gl_context_egl.cc:84] Successfully initialized EGL. Major : 1 Minor: 5
I20220725 20:54:06.374799 358149 gl_context.cc:335] GL version: 3.2 (OpenGL ES 3.2 NVIDIA 465.19.01)
I20220725 20:54:06.375151 358140 demo_run_graph_main_gpu.cc:68] Initialize the camera or load the video.
[ WARN:[email protected]] OpenCV | GStreamer warning: Cannot query video position: status=0, value=-1, duration=-1
I20220725 20:54:08.345331 358140 demo_run_graph_main_gpu.cc:89] Start running the calculator graph.
I20220725 20:54:08.349308 358140 demo_run_graph_main_gpu.cc:94] Start grabbing and processing frames.
F20220725 20:54:08.349412 358144 collection.h:417] Check failed: id < EndId() (1 vs. 1) 
*** Check failure stack trace: ***
    @     0x55e6ef943a8e  google::LogMessage::SendToLog()
    @     0x55e6ef941eea  google::LogMessage::Flush()
    @     0x55e6ef943f8d  google::LogMessageFatal::~LogMessageFatal()
    @     0x55e6ef86b8f7  mediapipe::internal::Collection<>::Get()
    @     0x55e6ef894348  mediapipe::InputSidePacketHandler::SetInternal()
    @     0x55e6ef894ca7  mediapipe::InputSidePacketHandler::Set()
    @     0x55e6ef8935a6  mediapipe::OutputSidePacketImpl::SetInternal()
    @     0x55e6ef8937a5  mediapipe::OutputSidePacketImpl::Set()
    @     0x55e6ef8496b2  mediapipe::(anonymous namespace)::EnvGeneratorCalculator::Open()
    @     0x55e6ef88ca8c  mediapipe::CalculatorNode::OpenNode()
    @     0x55e6ef87193b  mediapipe::internal::SchedulerQueue::OpenCalculatorNode()
    @     0x55e6ef871c3b  mediapipe::internal::SchedulerQueue::RunNextTask()
    @     0x55e6ef8ae9e7  mediapipe::ThreadPool::RunWorker()
    @     0x55e6ef8aefc3  mediapipe::ThreadPool::WorkerThread::ThreadBody()
    @     0x7f6fd39c1450  start_thread
    @     0x7f6fd2fbcd53  clone
    @              (nil)  (unknown)

I appreciate any help on the subject. Thanks

There is another issue entry for the same issue as far as I see which is #3060 . But there is no solution either. Please help me finding solution for that problem since I've been working on it for some time but couldn't fine any solution yet. Thanks, best wishes.

I have found that execution crashes at line 99 of demo_run_graph_main_gpu.cc .

capture >> camera_frame_raw;

Code line where the execution crashes is above. But I don't get why it happens. I appreciate any idea on the subject. Thanks!

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you.

Closing as stale. Please reopen if you'd like to work on this further.

Are you satisfied with the resolution of your issue? Yes No

Closing as stale. Please reopen if you'd like to work on this further.

Are you satisfied with the resolution of your issue? Yes No

Hi. When the example app crashes at the line:

capture >> camera_frame_raw;"

I suspect that the webcam is not properly initialized on the workstation. Could you possibly verify that there is a webcam available and that it can be accessed through the "cv::VideoCapture" library?

The stack trace in the bug report looks like a separate problem. I see the error message: 

collection.h:417] Check failed: id < EndId() (1 vs. 1)" during:  mediapipe::OutputSidePacketImpl::Set() during:  mediapipe::EnvGeneratorCalculator::Open()

This seems to indicate that EnvGeneratorCalculator is having difficulty delivering the output side-packet it produces.  It's supposed to deliver the "environment" side-packet to two "mirrors", each consumed by a "SwitchContainer". At least one of those consumers seems to be unprepared to accept the "environment" side-packet. This looks like some sort of disconnect between the nodes in the calculator-graph-config. It looks like the problem occurred in "face_effect_gpu.pbtxt". Do other examples run ok?

Hi. When the example app crashes at the line:

capture >> camera_frame_raw;"

I suspect that the webcam is not properly initialized on the workstation. Could you possibly verify that there is a webcam available and that it can be accessed through the "cv::VideoCapture" library?

The stack trace in the bug report looks like a separate problem. I see the error message:

collection.h:417] Check failed: id < EndId() (1 vs. 1)" during: mediapipe::OutputSidePacketImpl::Set() during: mediapipe::EnvGeneratorCalculator::Open()

This seems to indicate that EnvGeneratorCalculator is having difficulty delivering the output side-packet it produces. It's supposed to deliver the "environment" side-packet to two "mirrors", each consumed by a "SwitchContainer". At least one of those consumers seems to be unprepared to accept the "environment" side-packet. This looks like some sort of disconnect between the nodes in the calculator-graph-config. It looks like the problem occurred in "face_effect_gpu.pbtxt". Do other examples run ok?

Hi I can run the legacy MediaPipe Hand Detection solution with GPU, but also I get the described error in the issue.

---

I hope that this does help. When I comment this line (line 417) in collection.h:

CHECK_LT(id, EndId());

The error log becomes this:

I20230419 22:14:13.654337 15534 demo_run_graph_main_gpu.cc:59] Initialize the calculator graph.
I20230419 22:14:13.676187 15534 demo_run_graph_main_gpu.cc:63] Initialize the GPU.
I20230419 22:14:13.923753 15534 gl_context_egl.cc:84] Successfully initialized EGL. Major : 1 Minor: 5
I20230419 22:14:13.959640 15547 gl_context.cc:342] GL version: 3.2 (OpenGL ES 3.2 NVIDIA 525.105.17)
I20230419 22:14:13.959749 15534 demo_run_graph_main_gpu.cc:69] Initialize the camera or load the video.
I20230419 22:14:14.321822 15534 demo_run_graph_main_gpu.cc:90] Start running the calculator graph.
I20230419 22:14:14.322244 15534 demo_run_graph_main_gpu.cc:95] Start grabbing and processing frames.
E20230419 22:14:15.182279 15534 demo_run_graph_main_gpu.cc:201] Failed to run the graph: Graph has errors: 
; Input side packet with id 1 was already set.

(The webcam also starts capturing frames because the LED next to it turns on)

Hello @optimaltaskin, We are upgrading the MediaPipe Legacy Solutions to new MediaPipe solutions However, the libraries, documentation, and source code for all the MediaPipe Legacy Solutions will continue to be available in our GitHub repository and through library distribution services, such as Maven and NPM.

You can continue to use those legacy solutions in your applications if you choose. Though, we would request you to check new MediaPipe solutions which can help you more easily build and customize ML solutions for your applications. These new solutions will provide a superset of capabilities available in the legacy solutions. Thank you

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

This issue was closed due to lack of activity after being marked stale for past 7 days.

Are you satisfied with the resolution of your issue? Yes No