Cannot run selfie segmentation in web worker

[kirill-konshin]

Please make sure that this is a solution issue.

System information (Please provide as much relevant information as possible)

• Have I written custom code (as opposed to using a stock example script provided in Mediapipe): N/A
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04, Android 11, iOS 14.4): Any
• MediaPipe version: 0.1.1632777926
• Bazel version:
• Solution (e.g. FaceMesh, Pose, Holistic): Selfie Segmentation
• Programming Language and version ( e.g. C++, Python, Java): JavaScript

Describe the expected behavior:

Solution works the same way as in main thread

Standalone code you may have used to try to get what you need :

I was using Webpack 5.x and npm install @mediapipe/selfie_segmentation.

// main thread
const worker = new Worker(new URL('./worker.ts', import.meta.url));

// worker.js
import { SelfieSegmentation } from '@mediapipe/selfie_segmentation';

const ss = new SelfieSegmentation({
    locateFile: (file) => `https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/${file}`,
});

ss.setOptions({
    modelSelection: 1,
});

export const processFrame = (image: ImageBitmap) =>
    new Promise((resolve, reject) => {
        ss.onResults((results) => {
            ss.onResults(null);
            resolve(results.segmentationMask);
        });
        ss.send({ image }).catch(reject);
    });

Other info / Complete Logs :

Looks like some DOM objects are used in the solution code:

Uncaught ReferenceError: document is not defined
    at new Xb (selfie_segmentation.js?1643:73:229)

Additionally, ss.send has type issue:

TS2322: Type 'ImageBitmap' is not assignable to type 'InputImage'.   Type 'ImageBitmap' is missing the following properties from type 'HTMLCanvasElement': captureStream, getContext, toBlob, toDataURL, and 280 more.  index.d.ts(19, 3): The expected type comes from property 'image' which is declared here on type 'InputMap'

Similar bugs:

• https://github.com/google/mediapipe/issues/2506
• https://github.com/google/mediapipe/issues/2574
• https://github.com/google/mediapipe/issues/3406 (marked as stale, should be revived)

[cy-moi]

Hi, I'm having the same issue with face_mesh 0.4.1633559619. Have you figured it out?

Update: I removed the && "ontouchend" in document at line 75 in face_mesh.js and it worked.

[kuaashish]

Hi @kirill-konshin, Did this above workaround mentioned by @cy-moi resolves this issue. Thank you!

[kirill-konshin]

@kuaashish I'm not going to patch obfuscated sources, I'd rather wait for a proper fix. Moreover, the workaround is for face mesh, and I'm having issues with selfie segmentation.

I have found a bunch of document usages, including a.i=document.createElement("canvas") and case 7:d.l=document.createElement("canvas"), which I possibly can replace with OffscreenCanvas, but again, this does not feel right to patch sources like that. And also document.createElement("script").

[cy-moi]

@kuaashish I'm not going to patch obfuscated sources, I'd rather wait for a proper fix. Moreover, the workaround is for face mesh, and I'm having issues with selfie segmentation.

I have found a bunch of document usages, including a.i=document.createElement("canvas") and case 7:d.l=document.createElement("canvas"), which I possibly can replace with OffscreenCanvas, but again, this does not feel right to patch sources like that. And also document.createElement("script").

Not every document usage is actually called. Face mesh also has more than one document usage. I patched the face and hands packages for now. I wonder if they open-sourced the mediapipe libraries so we can modify on our own.

[kirill-konshin]

@kuaashish @cy-moi the above fix did not help.

Once removed, new error pop up. The lib simply cannot work in worker.

selfie_segmentation.js?1643:51 Refused to execute script from 'https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/selfie_segmentation_landscape.tflite' because its MIME type ('application/octet-stream') is not executable, and strict MIME type checking is enabled.

Failed to execute 'importScripts' on 'WorkerGlobalScope': The script at 'https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/selfie_segmentation_landscape.tflite' failed to load.

importScripts of tflite file does not work. Has to be replaced with fetch maybe?

[cy-moi]

@kuaashish @cy-moi the above fix did not help.

Once removed, new error pop up. The lib simply cannot work in worker.

selfie_segmentation.js?1643:51 Refused to execute script from 'https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/selfie_segmentation_landscape.tflite' because its MIME type ('application/octet-stream') is not executable, and strict MIME type checking is enabled.

Failed to execute 'importScripts' on 'WorkerGlobalScope': The script at 'https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/selfie_segmentation_landscape.tflite' failed to load.

importScripts of tflite file does not work. Has to be replaced with fetch maybe?

I don't think this error is caused by the patch of the package. Because I just tried to use selfie_segmentation directly without removing anything from its bundle, and the same error popped up. Have you solved it?

[cy-moi]

Hey, I had a look at the API of selfie segmentation. I guess they probably have already made it asynchronized with the OnResult callback. So probably no need to put it in the worker? edited: async* typo

[kirill-konshin]

Hey, I had a look at the API of selfie segmentation. I guess they probably have already made it synchronized with the OnResult callback. So probably no need to put it in the worker?

Worker is needed to offload all actual work from main thread. I have rest of the processing there.

[cy-moi]

Hey, I had a look at the API of selfie segmentation. I guess they probably have already made it synchronized with the OnResult callback. So probably no need to put it in the worker?

Worker is needed to offload all actual work from main thread. I have rest of the processing there.

Well, the above error is caused by accessing files through workers. So, probably try to put initialization into the main thread for an easy solution.

[kirill-konshin]

Hey, I had a look at the API of selfie segmentation. I guess they probably have already made it synchronized with the OnResult callback. So probably no need to put it in the worker?

Worker is needed to offload all actual work from main thread. I have rest of the processing there.

Well, the above error is caused by accessing files through workers. So, probably try to put initialization into the main thread for an easy solution.

Worker thread has no access to main thread, so this will not work. The problem is that these files are loaded via importScript instead of fetch.

[cy-moi]

Worker thread has no access to main thread, so this will not work. The problem is that these files are loaded via importScript instead of fetch.

I suppose it would be hard to pass the object around. But importScripts should work with workers. It is only the mime type problem. I think for one have you tried to host your model file with modified types to bypass the mime type check? Because I tried to host the tflite model at localhost and changed the filename to js (also need to modify the file path in mediapipe's package) and then the error is gone. So it has no problem import scripts. Finally, I still don't see the reason for the onresult to be put in the worker as it's completely non-blocking. The official example didn't even await for initialize().

[cy-moi]

Plus have you tried this tfjs models including mediapipe

[phanikavi]

@kirill-konshin, I experienced the same problem with older versions of selfie segmentation. I do not see the problem with current version: 0.1.1671057942

[kirill-konshin]

There's a new issue when browser loads ``

selfie_segmentation.js?1643:71 Refused to execute script from 'https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/selfie_segmentation_landscape.tflite' because its MIME type ('application/octet-stream') is not executable, and strict MIME type checking is enabled.
next-dev.js?3515:20 MAIN: Error in loop 1673479330696 Error: Error in worker: Error: Failed to execute 'importScripts' on 'WorkerGlobalScope': The script at 'https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/selfie_segmentation_landscape.tflite' failed to load.

The code was:

const ss = new SelfieSegmentation({
    locateFile: (file) => `https://cdn.jsdelivr.net/npm/@mediapipe/selfie_segmentation/${file}`,
});

I went ahead and modified the code to take files locally, with proper MIME type:

const cache = {
    'selfie_segmentation_landscape.tflite': require('@mediapipe/selfie_segmentation/selfie_segmentation_landscape.tflite'),
    'selfie_segmentation_solution_simd_wasm_bin.js': require('@mediapipe/selfie_segmentation/selfie_segmentation_solution_simd_wasm_bin.js')
};

const ss = new SelfieSegmentation({
    locateFile: (file) => cache[file].replace('//', '/')
});

And following next.config.js:

module.exports = {
    webpack: (config, options) => {
        config.module.rules.push({
            test: /\.(binarypb|tflite|wasm|data|bin)/,
            type: 'asset/resource',
        });
        config.module.rules.push({
            test: /_bin\.js/,
            type: 'asset/resource',
        });
        return config;
    },
    async headers() {
        return [
            {
                source: '/_next/static/media/:file*.tflite',
                headers: [
                    {
                        key: 'content-type',
                        value: 'application/javascript',
                    },
                ],
                basePath: false,
            },
        ];
    },
};

And now I am getting:

SyntaxError: Invalid or unexpected token

So I have two questions:

1. Why importScripts is importing a tflite file? Does not seem right.
2. If it is right, then which mime type it has to be served with?

[Mazuh]

If someone could answer @kirill-konshin that'd be nice, I could at least create a custom patch or something. My solution was similar to what he was attempting.

And fyi, I also made a few attempts to initialize anything on main thread and pass it to worker, but no matter what I tried for this approach I always reached an issue like a circular dependency, a shared non-primitive variable, or even a non-serializable complex object.

[matiaslopezd]

Could be approach use the model directly on a web worker using tensorflow?

importScripts('https://cdnjs.cloudflare.com/ajax/libs/tensorflow/4.2.0/tf.min.js');

async function loadModel() {
  const model = await tf.loadGraphModel('https://storage.googleapis.com/mediapipe-assets/selfie_segmentation.tflite');
  return model;
}

let modelPromise = loadModel();

onmessage = async (event) => {
  const { imageData } = event.data;
  const tensor = tf.browser.fromPixels(imageData);
  const model = await modelPromise;
  const output = await model.predict(tensor.expandDims(0)).squeeze();
  const segmentationMask = output.gather([0], 2).greater(0.5);
  const segmentationData = await segmentationMask.data();
  const width = segmentationMask.shape[1];
  const height = segmentationMask.shape[0];
  postMessage({ segmentationData, width, height });
};

[kuaashish]

@kirill-konshin,

Our MediaPipe Tasks can run in web worker. Please revert back if you still encounter the similar error.

[kirill-konshin]

@kirill-konshin,

Our MediaPipe Tasks can run in web worker. Please revert back if you still encounter the similar error.

No it cannot.

Error in worker: Error: Failed to execute 'importScripts' on 'WorkerGlobalScope': The script at 'http://localhost:3000/_next/static/media/selfie_segmentation_landscape.eec7c3d6.tflite' failed to load.

Version 0.1.1675465747 still requests tflite file via importScripts. It has to do fetch instead.

                    if ("function" !== typeof importScripts) throw Error("solutions can only be loaded on a web page or in a web worker");
                    d = b.filter(function(n) {return void 0 === n.simd || n.simd && c || !n.simd && !c}).map(function(n) {return a.locateFile(n.url, a.ha)});
                    importScripts.apply(null, ea(d));

[baumshl]

Any updates on this major issue that @kirill-konshin is reporting. I'm blocked on that as well. Please resolve.

[kuaashish]

@kirill-konshin,

We are pleased to announce that our solutions have undergone an upgrade. Selfie segmentation has been integrated into our Image segmentation framework, in accordance with the guidelines provided on this page. Additionally, for the upgraded solution, we have discontinued comprehensive support for previous versions.

We kindly request your participation in trying out our new Task API. and the upgraded Image segmentation, which is designed to facilitate the Self Segmentation task in accordance with the guidance provided here.

These enhancements also include support for web workers. Also, We request that you inform us if you observe similar behaviour in the upgraded solution. Thank you

[baumshl]

@kuaashish - as indicated by @kirill-konshin the issue is visible even in the new packages using the Task API.

[dbaynard]

I believe the web worker issue is when trying to install module web workers. This comes from the use of importScripts in mediapipe, which is not supported in module web workers. I've been able to replace this part of the code (to create detectors, not for selfie selection, though) and it works (mostly) fine.

The relevant part of the code calls await importScripts(…) which means it doesn't rely on importScripts being called synchronously. Instead it can be changed to await import(…) with some subsequent handling of the additions to the window scope.

So the problem is: given the code could work with import rather than importScripts, would it be possible for mediapipe to call the correct function, depending on its worker context (module vs script)?

[github-actions[bot]]

This issue has been marked stale because it has no recent activity since 7 days. It will be closed if no further activity occurs. Thank you.

[kirill-konshin]

@baumshl @dbaynard new @mediapipe/tasks-vision works in web worker.

[kirill-konshin]

The only problem with updated solution is that it returns canvas/texture/intarray/floatarray. Previous solution returned bitmap which was very useful. Is hope to get a method like result.confidenceMasks[0].getAsImageBitmap().

[kuaashish]

@kirill-konshin,

Thank you for the update. We kindly request that you initiate a new feature request with comprehensive details regarding the return of bitmap as a legacy solution, along with its associated benefits. This will enable our team to gain a thorough understanding and facilitate its implementation.

Furthermore, regarding the current issue, we seek your confirmation to update its status to resolved and proceed with its closure if it is no longer a concern from your end.

[baumshl]

@baumshl @dbaynard new @mediapipe/tasks-vision works in web worker.

Awesome to hear! Thanks for the confirmation, @kirill-konshin !!

[kirill-konshin]

Created https://github.com/google/mediapipe/issues/4798 for imagebitmap

[vastava]

@cy-moi have you open-sourced your patch for facemesh?

[cy-moi]

@cy-moi have you open-sourced your patch for facemesh?

Hi, you can find it here. But I haven't been using it for a while now. I hope it is helpful.