[bug] Cannot render more than a few video streams on web with overlayed participant names.

[jezell]

If you try to render more than 8 participants in web, each with a label rendered on top for it's name, the flutter web renderer will break due to using too many platform views.

A new change has landed in flutter designed to address this:

https://github.com/flutter/engine/pull/45256

The web renderer should be modified so that it uses createImageBitmap to render each frame so that apps can render more than 8 video feeds without crashing the flutter renderer.

[wanjm]

how to use createImageBitmap? do we need to create one timer.interval to refresh the videos;

[jezell]

createImageBitmap allows you to render a texture in flutter based off of a DOM media element (for example an img or video element can be used as a source). Under the covers, it ties into the browser APIs that exist to allow videos and images to be loaded and rendered into WebGL textures, so the performance is as optimized as possible in the browser.

timer.interval probably isn't the best approach as it can have a bit of latency in browser implementations, scheduleFrameCallback should provide a tighter update interval that aligns to frame boundaries.

Having done something similar in the past with javascript and livekit and pixi.js, there might be a little work to get the browser to always render the frames. In our experience, the browser would skip rendering video frames if it thought the video was off screen. When we did this with pixi.js + webgl a year ago, we had to use alpha:0, pointer-events:none, on the video element so that the browser didn't stop rendering it (rather than display none or not putting it into the DOM at all).

[jezell]

@eyebrowsoffire may be able to provide some additional guidance on the proper API usage for video elements.

[jezell]

Calling scheduleFrameCallback in a loop of course could result in the texture getting updated a lot faster than the framerate of the video, here's an example of how it's done in pixi.js:

https://github.com/pixijs/pixijs/blob/6b480827aecbacb57a6f6b162131772bb1931d55/packages/core/src/textures/resources/VideoResource.ts#L27

While it checks if a frame is due on every frame tick, it does limit the FPS of the GPU updates so that it only updates when a new frame is due. Note they make use of the video specific call here for requestVideoFrameCallback which notifies when the next video frame update is available.

[wanjm]

thanks.

[jezell]

@wanjm

https://github.com/flutter/packages/pull/6286

See here for a PR I did for video_player_web to fix the same problem. Not too many lines of code.