Support partial frame buffers

[waldheinz]

I have a use case which involves a quite large VNC server, of which only a small viewport is visible for each client. The current rfbClient.updateRect only partly solves the issues involved, because

• the client still has to allocate memory for the full frame buffer, which is impractical in my particular case
• the resulting frame buffer can not be efficiently used as an OpenGL ES 2.0 texture, because neither glTexImage2D nor glTexSubImage2D allow to specify a "scanline stride", and PBOs are only available with GLES 3. And I'm not even sure of PBOs would help here.

Because of this, I'd like to have libvncclient to support these use cases. Heck, I'd even implement it myself, but I'd like to get some input on how it could be done to increase the chances of having it merged upstream. :-)

My initial intention on how this might work is:

• there are additional fields for x and y offset in rfbClient (called offsetX and offsetY or whatever; set to 0 by default)
• rfbClient's width and height are no longer required to equal the server's frame buffer size, but can be smaller. Updates outside the viewport are clipped.
• rfbClient.frameBuffer is required to provide storage for (width - offsetX) * (height - offsetY) worth of pixels (instead of width * height as it stands currently)
• the decoders have to be adapted to honor the offsets, and this is the hard part:
  • the spec is not explicit about this, but it might be that a server references data in a copyrect update which the client has never requested but the server sent anyway. Currently, this is not a problem because the full frame buffer is available at client side anyway, but this is not the case when using the offsets / clipping. So I guess copyrect can not reliably work with such clients.
  • some decoders might be hard to refactor to match the changed semantics, especially if there is no server at hand to test it
  • in essence I'd like to have that only a subset of codecs is used when this offset / clipping stuff is in action, so it might work out to introduce a new function which enables this mode, which filters the currently active set of codecs so only ones where support was implemented are left, and issues a call to SetFormatAndEncodings on it's own behalf.
  • initially, this mode of operation would be implemented for raw and hextile encodings, because raw is required anyway and hextile offers sweet performance for my particular use case. Probably rre and corre, but I don't think I'll touch tight and the like.

Any thoughts on this?

[dscho]

Because of this, I'd like to have libvncclient to support these use cases. Heck, I'd even implement it myself, but I'd like to get some input on how it could be done to increase the chances of having it merged upstream. :-)

I hope that all public records of my activity show that I am easy to work with (if a bit demanding, but then more so on myself than on others)...

My initial intention on how this might work is [...]

Sounds all good!

A couple of specific comments:

it might be that a server references data in a copyrect update which the client has never requested but the server sent anyway.

I am not inclined to jump through hoops to support that use case... it is a bug in the server to send a copyrect if the client said it cannot handle it.

Or do you want to support the reduced client viewport in conjunction with the copyrect? I do not think that will work... we cannot copy what we do not have...

some decoders might be hard to refactor to match the changed semantics, especially if there is no server at hand to test it

This might be the perfect excuse to finally get a better test framework into place. At the moment, we have a really hard time to test things because we basically expect everything to go through sockets. But that does not need to be the case! We could easily open file descriptors for testing and use threads (or fork() on non-Windows, CreateProcess() on Windows) to

• start a server offering communication through pipes
• start a client testing specific things in a deterministic fashion

in essence I'd like to have that only a subset of codecs is used when this offset / clipping stuff is in action

That makes sense. The keyword is "initially": if we have a nice and easy way to run tests in-process basically with a client and a server talking to each other between two threads, we can ask for specific codecs and test the heck out of them. Of course, a proper audit is required at some stage but it would be better to have the tests first (because it makes it easier to find, reproduce and fix bugs that way).