Video pausing when browser is in the background

[peterzanetti]

I have implemented this my app, but when the browser/tab goes into the background, the video is pausing. How I can overcome this? Thanks.

[Volcomix]

The video is not supposed to be paused but this might be symptomatic of rendering timers being throttled in background tabs, which could introduce a lag effect (https://developers.google.com/web/updates/2017/03/background_tabs#background_timer_alignment).

As a workaround, you could try to play an almost inaudible sound with Web Audio API.

[jpodwys]

I can confirm that playing nearly silent audio is currently the best option--it prevents throttling (unless you're using requestAnimationFrame).

Unfortunately, this only works in Chromium browsers (not in Firefox or Safari). As a result, I typically only offer canvas-based rendering in Chromium browsers.

[peterzanetti]

Quite a bummer. The workaround is not working for me yet. Also was hoping to get this working in Safari.

[mikan3rd]

I solved the problem by using setTimeout instead of requestAnimationFrame.

[ArmorDarks]

It doesn't work in the background since requestAnimationFrame is used to read pixels data and is a driver behind rendering cycles. RAF is completely stopped when the tab is not visible. That's intended optimization browsers apply.

[bukharin]

Replacing RAF within runNotThrottledMacroTask may be a workaround:

let taskWorker: Worker;
export function runNotThrottledMacroTask(func: () => void) {
  if (!taskWorker) {
    taskWorker = new Worker(URL.createObjectURL(
      new Blob([`onmessage = function() { postMessage(1); };`], {type: "application/javascript"})));
  }
  const listener = () => {
    taskWorker.removeEventListener("message", listener);
    func();
  };
  taskWorker.addEventListener("message", listener);
  taskWorker.postMessage(1);
}

[benbro]

@bukharin don't you need a timeout inside the worker? Otherwise it'll just render as fast as it can.

[bukharin]

@benbro right! I think we need 2 workers like this:

1. Render loop worker with setTimeout for required FPS
2. GPU image upload awaiter without setTimeout here and await image uploading here. Otherwise we run inference on previous camera frame - you can see that while movng in demo - inference delays from your body

[benbro]

It's a bit hard to follow. Do you have a complete example?

We should all start the Chrome bug: https://bugs.chromium.org/p/chromium/issues/detail?id=1343635#c27

[bukharin]

I made a PR to fix the problem

[Volcomix]

Thank you for the comments and your contribution @bukharin. The timers through web workers are interesting.

I have mixed feelings regarding this issue though:

• on one hand this is an interesting issue which is worth investigating
• on another hand, this demo doesn't involve WebRTC and I don't think we have any way to actually see the issue happening in this demo (hence technically this is not an issue in this project).

IMO the best way to work on this would be to have a proper demo involving WebRTC to highlight the issue, experiment on a fix (or several fixes) and see how it behaves once fixed. However adding WebRTC into this project would require at least a server for the signaling channel (or a trick like this one 😉) but that would make this demo more complex.

Regarding the async pixels download, this has been added on purpose to increase the frame rate of the demo. Making it sync will introduce an FPS drop (most probably noticeable on mobile) but if you want more accurate results at lower frame rate, the proper solution would be to revert this change and to use gl.readPixels() instead of readPixelsAsync().

This week is quite loaded on my side so I'll try to take a deeper look at this during the upcoming week-end.

[bukharin]

@Volcomix I agree that demo doesn't involve WebRTC, but users takes this demo and your pipeline to reuse it in own projects and face the background problems. Replacing RAF to WebWorkers won't make this demo worse, it doesn't affect resulted FPS

I think that we should continue to use readPixelsAsync to avoid long running tasks that blocks UI thread. But we need to await (using webworkers) the result to ensure actual camera frame before running inference (I wrote about this bug in my PR)

[Volcomix]

I tested on the same mobile that I used for other performance tests (Pixel 3) with the fastest (default) parameters. And it seems that the FPS is affected by the change:

• with the current main of this repo: 60 fps - 1ms for resizing stage
• with #50: 40 fps - 10ms for resizing stage
• by removing the await in front of readPixelsAsync() from #50: 73 fps - 1ms for resizing stage

You are right that await readPixelsAsync() is better than gl.readPixels() because indeed it won't block the main thread (didn't understand that point at first). However, given the FPS drop that it introduces I think we should allow switching the await on and off through the UI (and comments in the code), so that people copying this could do the right choice depending on their need, to optimize for visual result or for the framerate.

Regarding the setTimeout in a web worker, most of my concerns are the ones mentioned in this issue that @benbro linked: https://bugs.chromium.org/p/chromium/issues/detail?id=1343635#c9.

I think we should at least handle the max FPS of this solution to prevent this demo to eat more CPU/GPU resources than needed to achieve a specific FPS. This would mean:

• adding a parameter in the UI with a target FPS (pre-filled at 60 fps)
• handling the time already elapsed in the pipeline to remove it from the timeout of the next run (I think this could be used as inspiration: https://github.com/chrisguttandin/worker-timers-worker/blob/master/src/helpers/timer.ts#L28, it also takes into account the time elapsed between posting the message from the main thread and receiving it into the worker)

[Volcomix]

Thanks a lot for you help on this @bukharin, that is very appreciated. af46ab9d1017d03ff1c519f395b996ee21006957 fixes the issue and is highly inspired from your PR.

Please anyone let me know if anything is going wrong with the fix/workaround. I already noticed that the FPS is more unstable and more difficult to predict but seems to be the price of having it working when the tab is in background.