Standalone tflite module release?

[saghul]

Hey there!

I was wondering if you'd consider releasing the bundled tflie module as standalone in for example. This would make it easier for those using to get updates, instead of having to vendor the files and check in here every once in a while to see the improvements you've made :-)

Thanks again for a great project!

[Volcomix]

Hey @saghul!

Yes I think I could release a standalone version of the tflite module. Just need to find the best way to include it in this repo (or another one?) and to define its interface. Also this hardcoded buffer size and the fact that some custom operators are strongly coupled to its usage with Google Selfie Segmentation model are things that I need to figure out how to make it clear (or customizable? not sure that's possible) in the package name/description/usage notes.

[saghul]

Hey!

Just need to find the best way to include it in this repo (or another one?) and to define its interface.

Maybe it could be another one and you can just depend on it?

Also this hardcoded buffer size and the fact that some custom operators are strongly coupled to its usage with Google Selfie Segmentation model

Interesting. We recently changed the model we use in Jitsi Meet from the initial Google Meet one to the now proper open source one and things Just Work (TM). Maybe we got lucky?

I'm curious now, can you share what parts are specific to the google selfie model? Thank you!

[Volcomix]

I'm curious now, can you share what parts are specific to the google selfie model?

Should be only

• this custom operator: https://github.com/Volcomix/virtual-background/blob/674aee05879e993189ccd12b439cf08b2221a80e/tflite/BUILD#L20
• and the buffer size which depends on the model size: https://github.com/Volcomix/virtual-background/blob/674aee05879e993189ccd12b439cf08b2221a80e/tflite/tflite.cc#L13

We recently changed the model we use in Jitsi Meet from the initial Google Meet one to the now proper open source one

I'm also becoming curious there. Is it the ML Kit Selfie Segmentation one which is available on 144x256 input and up? If yes and if you don't need the more optimized (but less accurate) 160x96 model, maybe should you consider switching to TFJS or MediaPipe which are now embedding a tflite module similar or maybe even better than the one in this repo?

• https://github.com/tensorflow/tfjs-models/tree/master/body-segmentation/src/selfie_segmentation_mediapipe
• https://google.github.io/mediapipe/solutions/selfie_segmentation.html#javascript-solution-api

[Volcomix]

Maybe we got lucky?

If you are now using ML Kit Selfie Segmentation, the only thing which could make it incompatible with the tflite module is the model size. And as the demo in this repo also handles this model, I increased the buffer size to make it work IIRC.

[saghul]

I'm curious now, can you share what parts are specific to the google selfie model?

Should be only

• this custom operator: https://github.com/Volcomix/virtual-background/blob/674aee05879e993189ccd12b439cf08b2221a80e/tflite/BUILD#L20

• and the buffer size which depends on the model size: https://github.com/Volcomix/virtual-background/blob/674aee05879e993189ccd12b439cf08b2221a80e/tflite/tflite.cc#L13

We recently changed the model we use in Jitsi Meet from the initial Google Meet one to the now proper open source one

I'm also becoming curious there. Is it the ML Kit Selfie Segmentation one which is available on 144x256 input and up?

If yes and if you don't need the more optimized (but less accurate) 160x96 model, maybe should you consider switching to TFJS or MediaPipe which are now embedding a tflite module similar or maybe even better than the one in this repo?

• https://github.com/tensorflow/tfjs-models/tree/master/body-segmentation/src/selfie_segmentation_mediapipe

• https://google.github.io/mediapipe/solutions/selfie_segmentation.html#javascript-solution-api

Interesting! We are using that model indeed. Thanks for the insight, I'll look into trying to consume TFJS and let's see how that goes.

Appreciate the detailed response!