noVNC Specify Multi-touch Support

At www.dizmo.com, we're developing an interface (of things), and decided to go for VNC and noVNC to enable browser support for our software, which relies as well on multi-touch support. However, noVNC does not seem to have such a feature enabled.

Hence, would it be possible to have multi-touch integrated? We would be willing to also pay the developer who integrates this feature.

I've issued a bounty of 240 DAI (a stable-coin corresponding to 240 USD) to have this issue not solved, but fully specified. Please have a look at: https://gitcoin.co/issue/novnc/noVNC/944/1047

My goal with this bounty would be to have exhaustive specification, which would allow me to open a follow-up of issue where then the actual multi-touch support is implemented. To claim the bounty, please provide a pull request with a corresponding Markdown document containing the specification.

Please see also our video gallery at https://www.dizmo.com/video-gallery/, where we display some of our use cases and where we plan to use multi-touch. In our case noVNC is running within a Docker container, hence the specification should be compatible with such a context.

NOTE: The bounty is structured as a contest with an expiration date of two weeks.

Oct 26 '17 09:10 hsk81

This would mean an extension of the RFB protocol, it is not trivial work.

Oct 27 '17 10:10 samhed

Is it maybe possible to provide an estimate (in terms of developer hours) w.r.t. to this RFB protocol extension? I'm not familiar with RFB, hence the question: why would multi-touch imply such an extension?

Dec 14 '17 12:12 hsk81

If you mean that touch events should be passed to the desktop or application in the remote session, then we do need an extension to the protocol. The protocol only allows regular mouse or keyboard input at the moment.

No I can't give you an estimate. Estimates never work out well.

Dec 14 '17 23:12 samhed

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 240.0 DAI (240.0 USD @ $1.0/DAI) attached to it.

If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
Want to chip in? Add your own contribution here.
Questions? Checkout Gitcoin Help or the Gitcoin Slack
$20,949.55 more funded OSS Work available on the Gitcoin Issue Explorer

Aug 28 '18 10:08 gitcoinbot

@samhed @CendioOssman @DirectXMan12 @kanaka I've issued a bounty of 240 DAI (a stable-coin corresponding to 240 USD) to have this issue not solved, but fully specified. Please have a look at:

https://gitcoin.co/issue/novnc/noVNC/944/1047

My goal with this bounty would be to have exhaustive specification, which would allow me to open a follow-up of issue where then the actual multi-touch support is implemented. To claim the bounty, please provide a pull request with a corresponding Markdown document containing the specification.

Please see also our video gallery at https://www.dizmo.com/video-gallery/, where we display some of our use cases and where we plan to use multi-touch. In our case noVNC is running within a Docker container, hence the specification should be compatible with such a context.

NOTE: The bounty is structured as a contest with an expiration date of two weeks.

Aug 28 '18 10:08 hsk81

@hsk81 Which type of specification are you leaning towards? RFB has been extended as a specification to support multitouch inputs, pressure sensitivity, and more via RFB GII standard extension in at least two major implementations that have code associated with them, starting here: https://groups.google.com/forum/#!topic/tigervnc-devel/Zupu-o6JDNI It appears in TurboVNC you have Remote Wacom Tablet mode which uses standard messages, as well as in TigerVNC. https://github.com/TurboVNC/turbovnc/blob/2e0a0fbad308b3fb53695e472679cd9b5024d48b/unix/Xvnc/programs/Xserver/hw/vnc/init.c#L974

However, these implementations have limitations. A correct proposal in my mind would specify equations for ellipsis for each contact point and its coordinates. An application of this would then natively identify gestures, for example on a tablet you might expect two fingers to zoom but one handed phone you might expect pinch to zoom.
The same messages used in RDP could be added to the GII format as well: https://msdn.microsoft.com/en-us/library/hh554669.aspx Please let me know your feedback, thanks in advance

Sep 10 '18 20:09 oogetyboogety

@mul1sh yes, it is still open! Please note that I'm looking for a specification only (for now at least).

Sep 18 '18 07:09 hsk81

@oogetyboogety The issue is that I'm myself not experienced enough w.r.t. multi-touch support in general. So your question about

Which type of specification are you leaning towards?

Is rather difficult to answer for me. What we have is the following software stack:

our own cross-platform QT based application, which has multi-touch support;
we have embedded it into a debian:jessie based docker container;
the container runs x11-xserver, tigervnc and noVNC;

We don't have multi-touch support with noVNC, and the goal of the specification would be to clearly define a roadmap to have multi-touch implemented: This may or may not involve configuring or extending other software (like x11-xerver or tigervnc) besides noVNC.

But obviously we do not want to reinvent the wheel, and have the desired minimal functionality with as little effort as possible.

RFB has been extended as a specification to support multitouch inputs, pressure sensitivity, and more via RFB GII standard extension in at least two major implementations that have code associated with them, starting here: tigervnc-devel. It appears in TurboVNC#L974 you have Remote Wacom Tablet mode which uses standard messages, as well as in TigerVNC.

So, if we could bring in multi-touch support to noVNC using somehow the work already done for RFB (GII) and TigerVNC then great, let's do that: Here, I would like a specification (more like a roadmap), which would allow me to clearly judge if it is feasible and how long it would take to have multi-touch for noVNC.

However, these implementations have limitations. A correct proposal in my mind would specify equations for ellipsis for each contact point and its coordinates. An application of this would then natively identify gestures, for example on a tablet you might expect two fingers to zoom but one handed phone you might expect pinch to zoom. The same messages used in RDP could be added to the GII format as well: https://msdn.microsoft.com/en-us/library/hh554669.aspx

I'm not interested to create new specifications (or improve existing ones), if RFB has already such a specification in place. I would prefer an existing specification.

Sep 18 '18 09:09 hsk81

@oogetyboogety - are you still working on this issue? Just checking in :)

Nov 23 '18 17:11 frankchen07

@oogetyboogety actually, yes: please see my most recent post above.

Nov 23 '18 18:11 hsk81

Issue Status: 1. Open 2. Cancelled

The funding of 240.0 DAI (240.0 USD @ $1.0/DAI) attached to this issue has been cancelled by the bounty submitter

Questions? Checkout Gitcoin Help or the Gitcoin Slack
$51,515.61 more funded OSS Work available on the Gitcoin Issue Explorer

Jan 25 '19 21:01 gitcoinbot

I looked into the possibility of implementing this in TurboVNC, since it already implements remote Wacom tablet support using the GII RFB extension. As has been pointed out above, GII does not currently accommodate multi-touch events, and it's unclear to me how those events should be implemented on the wire. I would need to develop a better understanding of how multi-touch is implemented in X Input, but since I don't have any devices (including the various Wacom tablets at my disposal) that generate such events, I don't have any way to develop that understanding at the moment.

Jan 18 '21 22:01 dcommander