wayvnc doesn't work with nvidia drivers, because neatvnc doesn't support 24bit pixel input

[DanielGibson]

I tried using cage (patched) and Sway (running headless with WLR_BACKENDS=headless) with wayvnc on my System (XUbuntu 22.04, Geforce 3060Ti, nvidia-driver 510.73.05-0ubuntu0.22.04.1). When connecting with a VNC viewer (tried TigerVNC and xtightvncviewer), it connects and immediately disconnects without a real error; wayvnc doesn't log an error either.

I debugged this and found hat that if fails in neatvnc's send_server_init_message(), at

	int rc = rfb_pixfmt_from_fourcc(&msg->pixel_format, fourcc);
	if (rc < 0)
		goto pixfmt_failure;

rfb_pixfmt_from_fourcc() fails because fourcc is 0x34324742 aka 'BG24' aka DRM_FORMAT_BGR888, which is a 24bit format (i.e. each pixel is 3 bytes, no X or A channel). (Side-Note: Probably that code should log an error, like the other error cases in the same function do)

This format comes from libwlroots: https://gitlab.freedesktop.org/wlroots/wlroots/-/blob/0.15.1/types/wlr_screencopy_v1.c#L539 and then is passed via wayvnc's screencopy code (screencopy_buffer() and screencopy_buffer_doen()) and probably some other intermediate steps I forgot to neatvnc. wlroots gets it in gles2_preferred_read_format(): https://gitlab.freedesktop.org/wlroots/wlroots/-/blob/0.15.1/render/gles2/renderer.c#L406-419 So in the end the problem is that nvidia's driver returns RL_RGB for glGetIntegerv(GL_IMPLEMENTATION_COLOR_READ_FORMAT, &gl_format); with an alpha size of 0 (and type GL_UNSIGNED_BYTE), which then gets translated to DRM_FORMAT_BGR888, which neatvnc doesn't like.

When I patch gles2_preferred_read_format() to always return DRM_FORMAT_XBGR8888 (nvidia might not "prefer" that, but it's supported apparently), wayvnc works as expected (except for input problems with cage, but that's probably a cage problem and I'll debug that separately).

Of course patching libwlroots is not a great solution, and this particular hack might even fail with other drivers (that might not support DRM_FORMAT_XBGR8888), so it would be best if wayvnc/neatvnc could support 24bit pixel formats as well.

I wasn't sure whether to report this as a wayvnc or neatvnc bug, but in abstract it might be fine for neatvnc not to support 24bit pixel input, but with the way wayvnc uses it it's problematic.

[any1]

Reading 24 bit pixel buffers can be implemented. Alternatively, buffers with unsupported pixels formats could be resampled, but that would require extra copying. Currently, a buffer will be resampled if it needs to be rotated, but the resampler could also be employed to do pixel format conversion.

[maztheman]

This will be resolved soon.

[maztheman]

Tight and raw should work with Nvidia drivers, but you'll have to build neatvnc and wayvnc from master branch.

EDIT: and now zrle also works.