mediapipe icon indicating copy to clipboard operation
mediapipe copied to clipboard

PoseLandmarker's segmentation is blank when using delegate: GPU

Open cristobalbahe opened this issue 10 months ago • 0 comments

Hello everyone, I am trying to debug the segmentation mask result of PoseLandmarker, but the result texture is blank. I initialize my vision task as the following

const vision = await FilesetResolver.forVisionTasks(
  "https://cdn.jsdelivr.net/npm/@mediapipe/tasks-vision@latest/wasm"
);
const poseLandmarker = await PoseLandmarker.createFromOptions(vision, {
  baseOptions: {
    modelAssetPath: "/pose_landmarker_full.task",
    delegate: "GPU",
  },
  runningMode: "VIDEO",
  outputSegmentationMasks: true,
});

Then I am running a renderLoop as such:

function renderLoop() {
  let startTimeMs = performance.now();
  if (video.currentTime !== lastVideoTime) {
    poseLandmarker.detectForVideo(video, startTimeMs, (result) => {
      if (result.segmentationMasks.length) {

        gl.viewport(0, 0, width, height);
        gl.useProgram(program);
        gl.activeTexture(gl.TEXTURE0);
        gl.bindTexture(
          gl.TEXTURE_2D,
          result.segmentationMasks[0].getAsWebGLTexture()
        );

        if (debug) {
          console.log(result.segmentationMasks[0].getAsFloat32Array());
        }
      }
    });
    lastVideoTime = video.currentTime;
  }

  requestAnimationFrame(() => {
    renderLoop();
  });
}

My WebGL canvas which is just rendering the WebGL Texture is not showing anything, so for debugging purposes I console logged result.segmentationMasks[0].getAsFloat32Array() to see if I was getting any values, but it turns out it's a 0 filled Float32Array.

I changed the delegate to CPU to see if I got a different result and, indeed, I got non 0 values inside the float32array, so I was wondering how could I get the segmentation mask with GPU as delegate.

Thank you so much in advance

cristobalbahe avatar Sep 04 '23 16:09 cristobalbahe