Problem when using mediapipes from inside a worker loaded as a module

[martenrichter]

Have I written custom code (as opposed to using a stock example script provided in MediaPipe)

None

OS Platform and Distribution

Web

MediaPipe Tasks SDK version

master branch

Task name (e.g. Image classification, Gesture recognition etc.)

Image classification but should apply to all tasks

Programming Language and version (e.g. C++, Python, Java)

Javascript

Describe the actual behavior

When loading the code I get an importScript exception

Describe the expected behaviour

That it also works from workers loaded as module

Standalone code/steps you may have used to try to get what you need

It is straightforward to reproduce. Just try to load it from a worker, which is loaded as a module, so something like this:

import { ImageSegmenter, FilesetResolver } from '@mediapipe/tasks-vision'
  async initSegmenter() {
    return ImageSegmenter.createFromOptions(
      await FilesetResolver.forVisionTasks(
        new URL(`../node_modules/@mediapipe/tasks-vision/wasm`, import.meta.url)
          .pathname
      ),
      {
        baseOptions: {
          modelAssetPath: new URL(
            './models/selfie_segmenter_landscape.tflite',
            import.meta.url
          )
        },
        canvas: this.canvas,
        outputCategoryMask: true,
        runningMode: 'VIDEO'
      }
    )
  }

If your worker is loaded as a worker, e.g.:

AVMediaPipe.workerInternal_ = new Worker(
  new URL('./avmediapipeworker.js', import.meta.url),
  {
    type: 'module'
  }
 )

You will get an error saying that importScripts is not supported. Note, as framework with bundlers such as vite, convert you commonJS module to ESM module, converting the worker to commonJS is not an option.

### Other info / Complete Logs

```shell
The origin is here:
https://github.com/google/mediapipe/blob/fde195e4a86298c12c300811b3ae239fd7b6238c/mediapipe/web/graph_runner/run_script_helper.ts.template#L13

I also tried to patch it and send a PR by replacing the first part of the code with:

if (typeof importScripts === 'function') {
     try {
        importScripts(scriptUrl.toString());
    } catch (error) { 
        if (error instanceof TypeError) {
          // importScripts does not work in module workers
          const module = await eval('import')(scriptUrl.toString());
          //@ts-ignore
          self.ModuleFactory = module.ModuleFactory
        } else {
          throw error
        }
    }
  } else {

However, I had terrible luck, as the typescript compiler in the project transpiles everything to commonJS and replaces the import with resolve, which is a dead end. (At least after a few hours, I did not find a suitable workaround.) What is the reason for compiling with TypeScript to CommonJS and converting it via rollup to an ES module, anyway? So, I am here asking for help or guidance on proceeding. Changing the overall compiling workflow and supplying a patch without asking would be a bad style.

[kuaashish]

Hi @martenrichter,

Could you please confirm whether the issue has been resolved on your end, or if you still require assistance from our end?

Thank you!!

[martenrichter]

Well, yes and no. I managed to patch the library and solve the issue by using a custom build of the package. The patch is the PR listed in this issue. Your colleague said he will integrate the changes manually, which makes sense as most changes come from upstream. I just tried an install (npm install @media-pipe/tasks-vision), and the version seems to be unchanged, not including the necessary changes. However, I have not checked all recent commits.

So, the problem persists, and I think a fix would benefit other users. (Of course, I will be happy to help.)

[kuaashish]

Hi @martenrichter,

Thank you for confirming. I will bring this to the attention of our team. Please allow us some time, and we will update you through the same thread once we have any further information available.

[martenrichter]

Hi @kuaashish, Sure, no problem; I have no hurry, as it works for me...

[arcinston]

any update on this , trying to use tasks-vision on web worker

[schmidt-sebastian]

We can try to apply your patch but our build pipeline is pretty complicated due to how our internal JS compiler is set up. While the external build should be fully functional, we are using a Google-internal compiler to further reduce our binary size.

[martenrichter]

The changes to the compilation were just necessary so that import() is not replaced with sth else ( if I remember correctly), so the question is if one can say to the compiler to leave it untouched?

[ricanteja]

Hi, any progress on this? Any workarounds? Thanks!

[martenrichter]

You can use the fork in my PR and compile it yourself. Or you can use the precompiled package from the fork, which I use as a workaround (but I have only done it for the vision package): https://github.com/fails-components/lectureapp/blob/master/packages/vision_pkg.tgz Of course, you should refer to the source code that led to the compilation. I have no intention of updating it, as I assume it would not take that long for it to be fixed upstream.

[rhwilburn]

I am using the package from @martenrichter, however there is an issue with the face mesh's when running from a worker where the worker doesn't understand: CanvasRenderingContext2D in the DrawingUtils contructor. See here: https://github.com/google-ai-edge/mediapipe/blob/c37721b9df27b03da34ff304696c2ed3c6d2b4cf/mediapipe/tasks/web/vision/core/drawing_utils.ts#L151

Understandably a worker would know about main thread rendering types like CanvasRenderingContext2D (as they instead use OffscreenCanvas equivalents).

My workaround is:

export class DrawingUtilsPatched extends DrawingUtils{
  constructor(args: any){
    self.CanvasRenderingContext2D = DrawingUtils;
    super(args);
  }
}

[martenrichter]

That is not surprising, as I use only the segmenter. I would guess upstream needs to change

    if (cpuOrGpuGontext instanceof CanvasRenderingContext2D ||
        cpuOrGpuGontext instanceof OffscreenCanvasRenderingContext2D) {

to

    if ((globalThis.CanvasRenderingContext2D  && cpuOrGpuGontext instanceof CanvasRenderingContext2D) ||
        (globalThis.OffscreenCanvasRenderingContext2D && cpuOrGpuGontext instanceof OffscreenCanvasRenderingContext2D)) {

for being saved. You can, of course, change it in the patched branch and build your own package inside the container; it is not that difficult. (Do not ask me how; I have already forgotten the building commands.) But I am glad, that I am not the only one with this scenario of using it inside a worker.

[joacopaz]

Any updates on this? It's been with no fix for a while. Any official patching will be much appreciated

[ppoitier]

Hello, can you tell us more about the current progress? It has been a while. It would be greatly appreciated if we could use task-vision in web workers so as not to freeze the main thread.

[Vizeo]

Being able to offload the image processing to a worker would be ideal. Is this still being worked on?