libvncserver icon indicating copy to clipboard operation
libvncserver copied to clipboard

VMWare mouse support

Open learn-more opened this issue 3 months ago • 8 comments

VMWare has a custom encoding for the cursor: https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst#vmware-cursor-pseudo-encoding

This is something that I am implementing in a tool I'm working on, would you be interested in me contributing it back?

If that is the case:

  • In what form would be preferred?
    • A protocol extension
    • Integrating in rfbclient (re-use HandleCursorPos / GotCursorShape or a new way, the format is slightly different)
  • Should there be a testcase for this
  • Should there be example code for this? (note, I could not find example code using the int8_t *rcSource, *rcMask; anywhere)

learn-more avatar Sep 17 '25 08:09 learn-more

Yes, contribution welcome.

What would speak against re-using HandleCursorPos / GotCursorShape ?

What would a testcase look like?

No example needed.

bk138 avatar Sep 17 '25 08:09 bk138

Yes, contribution welcome.

What would speak against re-using HandleCursorPos / GotCursorShape ?

What would a testcase look like?

No example needed.

HandleCursorPos can be used as-is, but the GotCursorShape will need to interpret data differently.

For 'normal' cursors, the rcSource and rcMask contain the data, where it looks like rcSource contains RGBA data, with the A zeroed out?

VMWare sends RGBA data with the alpha filled out, and no rcMask.

learn-more avatar Sep 17 '25 09:09 learn-more

HandleCursorPos can be used as-is, but the GotCursorShape will need to interpret data differently.

  • maybe it makes sense to introduce additional callbacks just to be explicit and not hide the fact that info might come from different sources.
  • or: can client API users distuingish otherwise. I.e. rcMask unset -> rcSource contains rgba ?

bk138 avatar Sep 17 '25 09:09 bk138

HandleCursorPos can be used as-is, but the GotCursorShape will need to interpret data differently.

* maybe it makes sense to introduce additional callbacks just to be explicit and not hide the fact that info might come
  from different sources.

* or: can client API users distuingish otherwise. I.e. rcMask unset -> rcSource contains rgba ?

Both could work. Another option would be that the code translates the vmware format to the existing rcMask/rcSource format?

This is really a question of api-design, so I think it makes sense for you to state your preference here. (I have not written the code yet, just made sure that I get valid data in a debugger).

For a testcase, I imagined that a stored message could be handled by the client code, and the resulting cursor be checked but I did not find other code doing stuff like that.

learn-more avatar Sep 17 '25 09:09 learn-more

Both could work. Another option would be that the code translates the vmware format to the existing rcMask/rcSource format?

This is really a question of api-design, so I think it makes sense for you to state your preference here. (I have not written the code yet, just made sure that I get valid data in a debugger).

Yes, unfortunately ;-) - Well the API needs a careful redesign later on anyway. Let's say that the callbacks are re-used and:

  • the code tries to translate incoming data to what users expext from the existing API
  • if that's not feasible, stores data in an additional rfbClient member and theres a Big Fat Warning in the API docs about this

For a testcase, I imagined that a stored message could be handled by the client code, and the resulting cursor be checked but I did not find other code doing stuff like that.

Yeah let's KISS and omit that.

bk138 avatar Sep 17 '25 09:09 bk138

Thanks for the fast responses, this is a great incentive to contribute!

learn-more avatar Sep 17 '25 10:09 learn-more

This is less compatible as I hoped. One of the cursors is fine (cursor type 1). The other one (cursor type 0) needs custom drawing code:

dst[i] = (dst[i] & and-mask[i]) ^ xor-mask[i];

So that is not translatable to the current image/mask.

When working with vmware, cursor type 1 is used for the arrow, and cursor type 0 is used for the 'I' cursor in text fields. (It is a black bar, but it is visible over a black line, probably due to the xor operation)

learn-more avatar Sep 19 '25 12:09 learn-more

Then it seems you need to go for approach b)

if that's not feasible, stores data in an additional rfbClient member and theres a Big Fat Warning in the API docs about this

bk138 avatar Sep 22 '25 17:09 bk138