yarn build:tflite fails

[edamlmmv]

I need to update the size of modelBuffer to use a bigger model of MediaPipe (MultiSegmentation).

I'm executing yarn build:tflite but it fails with the following error:

invalid registered toolchain '@local_config_python//:py_toolchain': error loading package '@local_config_python//': Unable to find package for @python//:defs.bzl: The repository '@python' could not be resolved: Repository '@python' is not defined.

Do you have an idea what could cause the issue?

[Volcomix]

I'm currently trying to fix the build. I managed to pass the Python error but now there are other errors popping related XNNPACK. You can find some updates in the branch fix-tflite-build-python. And you can see the latest logs here.

This is a tedious investigation and I won't have a lot of time to spend on it. So perhaps could you also experiment on a fork on your own.

Also now that there are other builds of tflite in tfjs and mediapipe repositories (see https://github.com/Volcomix/virtual-background/issues/38), have you considered trying to switch to them?

[edamlmmv]

I followed the most recent implementation of MediaPipe's image segmenter using the multi-segment model as it is the most accurate. However, the ImageSegmenter uses RequestAnimationFrame which freeze the stream transferred over WebRTC when changing tab so I need to do my own implementation similar to this repository which uses WebWorker and setTimeout.

I'll try to fix the XNNPACK issue. I'll report back if I find anything.

Thank you for taking time answering

[saghul]

Have tou tried using https://developer.mozilla.org/en-US/docs/Web/API/HTMLVideoElement/requestVideoFrameCallback instead of rAF ?

[edamlmmv]

1. I am not using (tflite.js | tflite.wasm | tflite-simd.js | tflite-simd.wasm) from this repository but instead vision_wasm_internal. (If you look into "@mediapipe/tasks-vision/wasm/" you'll find the files I'm talking about)

Inside of vision_wasm_internal, requestAnimationFrame is used. I don't think MediaPipe provide the C++ files from which vision_wasm_internal was compiled.

2. However, you've pointed out something I didn't realize. In my implementation I AM using requestVideoFrameCallback. requestVideoFrameCallback freezes as well when changing tabs

Therefore I may be able to make it work with my implementation if I try using a web worker + setTimeout similar to this demo. I thought I was done for because requestAnimationFrame was inside of vision_wasm_internal but I may have been looking at the wrong place.

I'll try and report back if I find anything

[edamlmmv]

I replaced requestVideoFrameCallback by setTimeout, added timerWorker.ts, added timerHelper and it worked! Thanks again for pointing that out saghul and for your time Volcomix.

-- videoFrameCallbackHandle = video.requestVideoFrameCallback(render); ++ timerWorker.setTimeout(render, 1000/30); // 30fps

If anybody is interested I could do a fork / branch where I implement vision_wasm_internal and MultiSegment model with ImageSegmenter into this demo.

[edamlmmv]

Err, should I re-open the issue since it is related to yarn build:tflite though?

[oGKarlin]

I replaced requestVideoFrameCallback by setTimeout, added timerWorker.ts, added timerHelper and it worked! Thanks again for pointing that out saghul and for your time Volcomix.

-- videoFrameCallbackHandle = video.requestVideoFrameCallback(render); ++ timerWorker.setTimeout(render, 1000/30); // 30fps

If anybody is interested I could do a fork / branch where I implement vision_wasm_internal and MultiSegment model with ImageSegmenter into this demo.

I would be interested in having that fork :)

[edamlmmv]

Kk I will work on it this weekend and report back. May take 2 weeks though.