[libvncclient] find a way to preset alpha data in framebuffer data

[willbprog127]

I have been using libvncclient for a while now in my project, but I still have not found a way to disable the alpha information on a client connection. I do not need or want the alpha information, so I am forced to go through the framebuffer and pick out the RGB data and then manually set the alpha to full so that my graphical toolkit will actually draw the image.

I am init'ing the connection like so (relevant code only):

vncClient = rfbGetClient(8, 3, 4);

vncClient->appData.encodingsString = "tight";

I am setting the alpha to full from the framebuffer like so (simplified from the original):

unsigned char* imageData = malloc(vnc->buffSize);

if (imageData == NULL)
    return;

// set alpha to full on 4 bytes-per-pixel
if (vnc->bytesPerPixel == 4) {
    for (int i = 0; i < vnc->buffSize; i++) {
        // set r, g, b data
        if (inc < vnc->bytesPerPixel) {
            vnc->imageData[i] = vnc->vncClient->frameBuffer[i];
            inc++;
        } else {
            // set alpha to full opacity
            vnc->imageData[i] = 255;
            inc = 1;
        }
    }
}
else
    memcpy(vnc->imageData, vnc->vncClient->frameBuffer, vnc->buffSize);

To improve app performance, I want to avoid doing this extra step. Is there any way to disable alpha on the connection? If it's a specific kind of encoding, what kind would you suggest and how do I set this? The documentation is not super clear about how this is performed.

Thanks! 👍

[bk138]

Hi Will, Sorry for the late reply! As far as I can remember, libvncclient apparently always sets the alpha info to zero, so you have to set the alpha info of your target framebuffer manually or have it pre-set to 255. Here https://github.com/bk138/multivnc/blob/master/src/VNCConn.cpp#L1376 is what I did back then in my viewer project.

If you'd be able to use vnc->framebuffer as a source for you graphical toolkit directly, you could skip the memcpy(), propably getting some speedup...

HTH, Christian

[willbprog127]

@bk138 Christian, thanks for the reply! 👍

I actually do skip memcopy() now. I have made some revisions since I last posted this issue.

What would be very helpful is if you could set the default alpha value per connection. I'm using FLTK with libvncclient and the drawing routines draw everything transparent (so not visible) from vnc->framebuffer because the alpha byte is 0 by default. If the alpha could be set, say in rfbGetClient to whatever you want, then the alpha wouldn't have to be set each time the client draws, saving a ton of processing time. This would be a nice dream. 😄

[bk138]

I can do some digging in the code, it's been a while since touched this. But should be possible!

[willbprog127]

Hello, it's been a while.

I have been experimenting with libvncclient 0.9.12 (from MacPorts) on macOS and now it appears I no longer need to set the alpha to 255. I looked through the commits but didn't see anything mentioning this issue. Was this issue resolved (in my case) by a commit for another issue?

Thanks! 👍

[willbprog127]

Greetings again, after a long time!

My message above was in error, the 'problem' is not resolved.

@bk138 Is there some way, through client->format, rfbInitClient() or something else, where I can tell the server I don't want any alpha? All I want is the RGB and not A. Am I just missing a setting or format request? It's kind of wasteful to push the alpha data over the wire when I don't even want it. Hopefully this question makes sense...

Thank you! 👍

[badda71]

Same issue here, I solved it using the following code. However, it would be nice if libvnc would be able to set the alpha itself during frame buffer update. This way we would not have to loop over the framebuffer twice and save some cycles ...

#define DUFFS_LOOP(pixel_copy_increment, width)	\
{ int n = (width+7)/8;				\
	switch (width & 7) {			\
	case 0: do {	pixel_copy_increment;	\
	case 7:		pixel_copy_increment;	\
	case 6:		pixel_copy_increment;	\
	case 5:		pixel_copy_increment;	\
	case 4:		pixel_copy_increment;	\
	case 3:		pixel_copy_increment;	\
	case 2:		pixel_copy_increment;	\
	case 1:		pixel_copy_increment;	\
		} while ( --n > 0 );		\
	}					\
}

static void handleFrameBufferUpdate_alpha (struct _rfbClient *client, int x, int y, int w, int h)
{
	int skip = (client->width - w) * 4; // 4 bytes per pixel (ARGB)
	unsigned char *buffer = client->frameBuffer + (y * client->width + x) * 4;
	DUFFS_LOOP ({
		DUFFS_LOOP({
			*buffer = 0xFF;
			buffer+=4;
		}, w);
		buffer += skip;
	}, h);
}

...
client->GotFrameBufferUpdate = handleFrameBufferUpdate_alpha;
...