noVNC icon indicating copy to clipboard operation
noVNC copied to clipboard

WebCodecs H.264

Open any1 opened this issue 7 months ago • 16 comments

This adds the Open H.264 encoding to noVNC, as defined here: https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#open-h-264-encoding

This was tested on the following browsers:

  • Firefox 127.0.2 with dom.media.webcodecs.enabled=true
  • Firefox Nightly 129.0a1 (2024-07-08)
  • Chromium 126.0.6478.126

I haven't run into any issues with Chromium, but Firefox keeps killing the decoder after a few frames because it thinks that the hardware decoder is too slow, so it wants to switch to software. However, after the switch, a new key frame is needed, so the decoder goes into a bad state and a reconfiguration is required, so this sequence of events is repeated indefinitely.

We can work around Firefox's issue by setting a frame duration for each frame, but that would be a hack. We don't know the frame duration for sure, and it is an optional parameter so Firefox should not rely on it.

I created an issue in bugzilla here: https://bugzilla.mozilla.org/show_bug.cgi?id=1906761

On the server side, I tested this against wayvnc.

WLR_BACKENDS=headless sway &
WAYLAND_DISPLAY=wayland-2 wayvnc -g

WayVNC can also run as a WebSocket server using the -w flag.

any1 avatar Jul 08 '24 22:07 any1