0.10.10 Holistic GPU delegation does not work

[clemenshimmer]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

Yes

OS Platform and Distribution

Ubuntu 22.04.3 LTS

MediaPipe Tasks SDK version

0.10.10

Task name (e.g. Image classification, Gesture recognition etc.)

Holistic Landmarker

Programming Language and version (e.g. C++, Python, Java)

JS WASM API

Describe the actual behavior

Task cannot run when GPU delegation is turned on

Describe the expected behaviour

Tasks runs with GPU delegation turned on

Standalone code/steps you may have used to try to get what you need

https://codepen.io/clemenshimmer/pen/rNREezm

To break the behaviour, simply uncomment the GPU delegation option.

Other info / Complete Logs

I tried to get the just released 0.10.10 holistic landmarker to run, as mentioned in the release notes:

Javascript Add Holistic Landmarker Web API

I've been a tiny bit cheeky and just walked back the path of the face landmarker task file linked in the docs until I landed on https://storage.googleapis.com/mediapipe-models/ which gave me the contents in a searchable XML file, that's how I found https://storage.googleapis.com/mediapipe-models/holistic_landmarker/holistic_landmarker/float16/latest/holistic_landmarker.task and just used it as the TS API suggested to. That seems to work, but only if GPU delegation is turned off, once it is turned back on the following error is thrown.

Run error (stack traces truncated)

Following operations are not supported by GPU delegate:

DEQUANTIZE: 

STRIDED_SLICE: Slice does not support shrink_axis_mask parameter. 

123 operations will run on the GPU, and the remaining 59 operations will run on the CPU.

E0223 11:50:16.711000 1927856 calculator_graph.cc:892] UNIMPLEMENTED: CalculatorGraph::Run() failed: 
Calculator::Open() for node "mediapipe_tasks_vision_holistic_landmarker_holisticlandmarkergraph__mediapipe_tasks_vision_face_landmarker_faceblendshapesgraph__mediapipe_tasks_core_inferencesubgraph__inferencecalculator__mediapipe_tasks_vision_holistic_landmarker_holisticlandmarkergraph__mediapipe_tasks_vision_face_landmarker_faceblendshapesgraph__mediapipe_tasks_core_inferencesubgraph__InferenceCalculator" failed: No support of const
=== Source Location Trace: ===
third_party/ml_drift/gl30/inference_context.cc:814
third_party/ml_drift/gl30/inference_context.cc:984
third_party/ml_drift/gl30/inference_context.cc:919
blaze-out/wasm-opt-ST-41541f88f73d/genfiles/third_party/mediapipe/calculators/tensor/inference_calculator_ml_drift_webgl_obfuscated.cc:288
blaze-out/wasm-opt-ST-41541f88f73d/genfiles/third_party/mediapipe/calculators/tensor/inference_calculator_ml_drift_webgl_obfuscated.cc:489
blaze-out/wasm-opt-ST-41541f88f73d/genfiles/third_party/mediapipe/calculators/tensor/inference_calculator_ml_drift_webgl_obfuscated.cc:119
third_party/mediapipe/framework/calculator_node.cc:560

Originally I recieved this error, but it seems this is some kind of error on my end as the minimal reproduction works - at least with GPU delegation disabled.

Initialization error (stack traces truncated)

No registered object with name: mediapipe::tasks::vision::holistic_landmarker::HolisticLandmarkerGraph; Unable to find Calculator "mediapipe.tasks.vision.holistic_landmarker.HolisticLandmarkerGraph"; Initialize was not ok [type.googleapis.com/mediapipe.StatusList='\n\xcb\x01\x08\x05\x12\xc6\x01No registered object with name: mediapipe::tasks::vision::holistic_landmarker::HolisticLandmarkerGraph; Unable to find Calculator \"mediapipe.tasks.vision.holistic_landmarker.HolisticLandmarkerGraph\"']
=== Source Location Trace: === 
./third_party/mediapipe/framework/deps/registration.h:218
third_party/mediapipe/framework/validated_graph_config.cc:230
third_party/mediapipe/framework/validated_graph_config.cc:350
third_party/mediapipe/framework/calculator_graph.cc:454
research/drishti/app/pursuit/wasm/graph_utils.cc:87

ERROR Error: Uncaught (in promise): Error: ValidatedGraphConfig Initialization failed.
No registered object with name: mediapipe::tasks::vision::holistic_landmarker::HolisticLandmarkerGraph; Unable to find Calculator "mediapipe.tasks.vision.holistic_landmarker.HolisticLandmarkerGraph"; Initialize was not ok; StartGraph failed
Error: ValidatedGraphConfig Initialization failed.
No registered object with name: mediapipe::tasks::vision::holistic_landmarker::HolisticLandmarkerGraph; Unable to find Calculator "mediapipe.tasks.vision.holistic_landmarker.HolisticLandmarkerGraph"; Initialize was not ok; StartGraph failed
  at Wo (vision_bundle.mjs:1:44941)
  at wh.setGraph (vision_bundle.mjs:1:46735)
  at wh.m (vision_bundle.mjs:1:118111)
  at vision_bundle.mjs:1:46413
  at _ZoneDelegate.invoke (zone.js:368:26)
  at Object.onInvoke (core.mjs:11083:33)
  at _ZoneDelegate.invoke (zone.js:367:52)
  at Zone.run (zone.js:129:43)
  at zone.js:1257:36
  at _ZoneDelegate.invokeTask (zone.js:402:31)
  at resolvePromise (zone.js:1193:31)
  at zone.js:1100:17
  at zone.js:1116:33
  at asyncGeneratorStep (asyncToGenerator.js:6:1)
  at _throw (asyncToGenerator.js:25:1)
  at _ZoneDelegate.invoke (zone.js:368:26)
  at Object.onInvoke (core.mjs:11083:33)
  at _ZoneDelegate.invoke (zone.js:367:52)
  at Zone.run (zone.js:129:43)
  at zone.js:1257:36

[ButzYung]

I can confirm the same issue. Holistic landmark works only if GPU delegation is disabled.

[ayushgdev]

Hello @ButzYung and @clemenshimmer Thanks for filing this issue. We have escalated this issue to the concerned team developing the JS APIs for MediaPipe. The issue is being investigated.

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[arpu]

still a problem with new 0.10.11

[Wei-1]

Still seeing the same issue in 0.10.12.

[arpu]

@schmidt-sebastian anything we can try/help?

[MuhammadAasharibNawshad]

seeing the same issue in 0.10.13rc.

[ButzYung]

Unfortunately GPU delegation is still not working on the latest v0.10.13. I wonder if there is a roadmap on when GPU will be supported on Holistic Web. Without GPU, web solution is barely usable for any real-time application.

[Wei-1]

Just tested with v0.10.15, the GPU support for Holistic Web is still not working. Edit: in WebWorker

[AmyangXYZ]

With 0.10.15 and 0.10.16, Holistic Web runs without error and the GPU delegation actually works, FPS increased from below 10 to 15? demo: https://mikapo.amyang.dev

[ButzYung]

With 0.10.15 and 0.10.16, Holistic Web runs without error and the GPU delegation actually works, FPS increased from below 10 to 15? demo: https://mikapo.amyang.dev

The FPS is above 20 judging from your demo. CPU-only Holistic was hardly over 10 in my own previous tests, so I bet GPU is working.

PS: Well it seems it still doens't work in my app (using 0.10.15, GPU delegation on but with the same errors as before)? Is there something special regarding your Holistic setup?

[AmyangXYZ]

With 0.10.15 and 0.10.16, Holistic Web runs without error and the GPU delegation actually works, FPS increased from below 10 to 15? demo: https://mikapo.amyang.dev

The FPS is above 20 judging from your demo. CPU-only Holistic was hardly over 10 in my own previous tests, so I bet GPU is working.

PS: Well it seems it still doens't work in my app (using 0.10.15, GPU delegation on but with the same errors as before)? Is there something special regarding your Holistic setup?

Nothing special actually, https://github.com/AmyangXYZ/MiKaPo/blob/261fd2b9229a99488e66d6790c9120ad8e053aab/src/Video.tsx#L181-L265