Feature/input emulation

[stheid]

I have created a proof of concept implementation to add mousepointer emulation to your tool. This is related to an issue i initially opened at reStream, but is much more well suited in your project https://github.com/rien/reStream/issues/37

• click on the rmview window emulates touch events on the remarkable (gestures also work)
• click+modifier on rmview window emulates digitizer events, allowing basic drawing.

The code is terrible and i would definetly not recommend to merge it :D

I wanted to show the feature and maybe discuss on how to cleanly reimplement it. Reverse engineering the touch and stylus where challenging enough for the first step :D

Please let me know what you think. (also sorry for all the diffs, but pycharm seems to disagree with your file formating ^^)

[bordaigorl]

Oh thanks! Yes my formatting is absolutely horrifying I know! I keep promising myself to run a linter and never do it. The author of the VNC server was also thinking of integrating the cursor in the server. We briefly discussed this in the tablet->computer direction (see here) but in principle the server could receive pointer updates too (I haven't tried).

I was thinking of adding key strokes actually to be able to flip pages...

[stheid]

Definetly keep me in the loop! I am now an expert of the RM2 touch and digitizer key codes :D so maybe i can help from the technical side here as well. I plan to write up my findings in form of a technical documentation, but i have to find time for it and many details are still not understood.

Please let me know if you are interested to integrate my work in the next weeks, than i would clean up my part and rebase it, such that i can make a proper PR.

Maybe one disclaimer, i am only owning an RM2 hence my input codes might not work properly on the RM1, but it seems that the they dont differ much (when reading arround in the other repositories)

Also let me know if you can confirm that everything is working. for you.

[bordaigorl]

Re testing: I will test it on my RM1 maybe this weekend. What I think could be useful is: emulating touch events on click and emulating arrow key presses to change page. One question: why did you need to reverse engineer it? The events seem to conform to the evdev format (I am no expert though). For example one could take the constants/code resolution functions from Python-evdev (although importing it as a full dependency would be overkill)

[stheid]

frankly i dont know a lot about evdev even though i came across it when looking at other code. Perhaps there is a much easier way to deal with it than

emulating the touch events to trigger page flip definetly makes sense but i still like the enhanced functionallity to also be able to do other stuff.

If you are right with evdev, i think it would indeed make sense to add it as a dependency, as using an existing library is certainly beneficial for robustness.

[bordaigorl]

Let me clarify: Python-evdev cannot be used as is because it only manipulates local devices afaict. The hint was to look there for the structs and codes so that you don't need to reverse-engineer them. To implement this feature I would still go with your approach of writing to the event files (although I would maybe not use sftp for that, but I'll have to experiment a bit). The page flipping can be triggered by emulating keystrokes, which is way easier than emulating swipes (if that's what you meant).

[stheid]

i am not 100% sure it can only be used on local device files and not on ssh mounted files

I have been looking at the stuff that has been documented so far in other projects, however the single commands for the stylus are still quite complicated and i had to try arround a lot to get the device working. touch was quite trivial actually. You can see how elaborate the codes are if you look at my code.

The RM2 has no physical keys! I afaik the only way to make page flipping is to emulate the touch gestures.

[stheid]

Have you had time to look at it in the mean time and are you interested to integrate it into the project?

[pl-semiotics]

Oh thanks! Yes my formatting is absolutely horrifying I know! I keep promising myself to run a linter and never do it. The author of the VNC server was also thinking of integrating the cursor in the server. We briefly discussed this in the tablet->computer direction (see here) but in principle the server could receive pointer updates too (I haven't tried).

I was thinking of adding key strokes actually to be able to flip pages...

Just to clarify---the vnc server supports digitizer, touch, and key events in the computer->tablet direction, independently of the other direction. I use this occasionally myself; feel free to ping me if any changes in the support would be useful.

[pl-semiotics]

(One advantage of using that is that it takes care of setting up a uinput device for the keyboard input on the rm2 where there is no native keyboard.)

[bordaigorl]

Ah @pl-semiotics that's amazing thanks, I did miss that! If I have a spare cycle in the weekend I'll give it a go!

[bordaigorl]

@stheid it is indeed a nice to have feature, but I would implement it by sending the events to the VNC server as @pl-semiotics suggested. Feel free to make an attempt at it if you feel like it, the rfb.py module should support this (I have to check)

[pl-semiotics]

@bordaigorl Ah, wonderful, glad I could clear that up! Feel free to ping me if you run into any trouble with the interface.

@stheid By the way, in case you were looking for the documentation around all of this, the kernel input system is documented nicely here. The evdev event interface (providing struct input_event and the various EVIO ioctls) is thankfully nowadays the main way that the kernel handles input. The documentation describes the various event codes and the multi-touch protocol (which is a bit more complicated than the wacom protocol, especially if one wants to to be able to inject extra touchpoints while not interfering with those that are actually being reported by the kernel---I admit that my solution has not been robustly tested in the case of touch input appearing both on the digitizer and the vnc connection, although it does attempt to allocate slots and tracking ids in such a way as to (hopefully) avoid those that the kernel chooses). The actual structs are defined in include/uapi/linux/input.h, and the various codes in include/uapi/linux/input-event-codes.h. The main difficulty with accessing evdev files remotely (they are device special files!) is the occasional need to use the EVIO ioctls, although one can often get by without them (I use them only for uinput device setup, device detection, and handling SYN_DROPPED events).

xochitl on the RM2 is clearly built from mostly the same code as the RM1 version, and therefore luckily has keyboard input handling left in it even though there is no actual keyboard on the device. This is nice since one can just add a virtual keyboard via uinput, which is what I do.

The code that the VNC server uses to handle interacting with input devices is here.

[stheid]

Definetly way better to implement it properly! I am also interested to help depending on my time. @pl-semiotics Very nice pointers to documentation. Exciting :)

[bordaigorl]

I implemented this in the pointer-events branch. Mouse events are translated to touch events, unless shift is pressed, in which case they are translated to pen events. Left/Right presses are forwarded allowing changing pages. I tested it on RM1, works like a charm! This required a refactoring that allowed me to implement a Pause/Resume feature which can be useful (for example if you need to tap in your pin or switch documents without showing your folders) so I threw this in too. If you could test it too I'd merge it in the main branch.

[bordaigorl]

@pl-semiotics I am noticing that sometimes buttons become unresponsive on the tablet (until next restart of xochitl) after using it with rmview (with the pointer-events version). My guess is that the killall that rmview is doing to kill the server before quitting is not giving the server enough time to properly release the input device. What's the proper way to terminate the server?

[bordaigorl]

@stheid @pl-semiotics could you test the pointer-events branch? I think I now got it working reliably, but wanted to check. A few iterations back, button input would randomly stop working on the device until xochitl was restarted. It hasn't happened since the last commit but I am not really sure what was going wrong.

[stheid]

Sry, didnt read. i will checkout and test your code within this week

[stheid]

drawing and clicking works great :) Somehow the gestures dont seem to work, but maybe i can figure out whats going on. Very nice. I think this PR can be closed and details are perhaps better discussed in the other PR?

tested on RM2

[stheid]

The thing with the gestures seems to be a little more severe:

• start rmview
• click once on the screen (with rmview)
• [ERROR] gestures dont work anymore (also not with the finger)

---

• error remains when rmview is stopped

workarrounds

• trigger input event with the stylus (getting "close" is enough") (seems to reset the inputs)

source of the error

connecting to the VNC server via remmina results in the same behaviour (click on the screen via the vnc client breaks the touch gestures)

hence i assume its related to the vnc server below and not an issue related to rmview itself.

button events

despite the fact that the rM2 does not have physical keys, your shortcuts for paging (ctl+left...) work perfectly (basically recreating the gestures in nice, however i want to point out again, the gestures are also broken when physically touching the device unless you reset the events with the stylus first.)

pausing and resuming

also seems to work like a charm :heart:

[pl-semiotics]

Oh dear, I apologize for not seeing this earlier. This is an interesting failure mode that I have not seen in my own testing; probably there is some slight error in the multitouch-protocol-b state machine that makes xochitl think that there is an extra finger down or some such, causing issues.

I haven't been able to easily reproduce this, so I think I need to look at input logs. I cross-compiled evtest for the RM1 and RM2; could you please run it, select the touchscreen input device (on my rm1, it's called cyttsp5_mt---it's not a wacom and not a keyboard, anyway), and then reproduce the issue and send me the logs?

@bordaigorl there shouldn't be anything terribly interesting w.r.t. termination (well, maybe don't kill it while it's in the middle of writing an event, but that shouldn't be too likely)---I also usually just kill it when done. By buttons, do you mean the ones on the touchscreen or the hardware ones? The touchscreen I can imagine (although not reproduce; I will try to debug it if I can get event logs), but I really don't see a reason why the hardware keys ought to be affected.

[stheid]

When i was talking about "button emulation" i was talking about the following:

The RM1 has physical buttons to go to next/previous page and close current file (3 in total) In the RM2 this functionallity is achieved by swipe gestures as i does not have physical keys. Wenn looking at rmviews source code ctrl + left should emulate the physical left key, however on the rm2 it does not exist. nevertheless the events are handled correctly despite the fact that this cannot be triggered on the remarkable normally. It seems that they kept the input events for software compatibility and hence the emulation works on the rm2.

That said it is very nice and not a bug of any sort. i was merely surprised that this would work at the rm2 as i would have expected it to ignore the events.

[stheid]

I haven't been able to easily reproduce this, so I think I need to look at input logs. I cross-compiled evtest for the RM1 and RM2; could you please run it, select the touchscreen input device (on my rm1, it's called cyttsp5_mt---it's not a wacom and not a keyboard, anyway), and then reproduce the issue and send me the logs?

To be clear:

• copy evtest file to rm2
• execute it in an ssh console
• touch the screen
• stop evtest
• copy log file
• remove evtest
• send it ?

or should i record anything when triggering events though rmview?

[pl-semiotics]

When i was talking about "button emulation" i was talking about the following:

Yes, I know. I was asking about the phrasing that @bordaigorl identified in https://github.com/bordaigorl/rmview/pull/40#issuecomment-770499411 .

I am glad the keyboard emulation is useful. Luckily, enough code is shared between xochitl on RM1 and RM2 that it still uses any keyboard devices that appear, so a uinput virtual keyboard can work.

or should i record anything when triggering events though rmview?

Sorry, I should have been more clear. Ideally:

• Copy evtest to the rm2
• Execute it and select the touchscreen, saving the output to a file
• Run rmview and trigger the physical-touchscreen-doesn't-work problem
• Try using the touchscreen
• Send me the file with the output of evtest

So that I can see the event sequence that rM-vnc-server is sending that is causing the issue in order to debug it.

(If it requires multiple attempts to reproduce the issue, please send the file containing just the events from the attempt that did cause the issue.)

[stheid]

haha, fortunately the bug is very consistent ^^. i will do in the next couple of days. Thanks for your cool work :)

[stheid]

I think the requested logs dont reflect anything about the bug :(

As said before, i can trigger the bug by only triggering the emulated stylus. As seen in the log, this causes actually no events at all on the touch device. touch-braking-gestures.log

Those are the two logs for playing out the gesture on the tablet.

1. while the bug is present (and gesture is actually ignored) touch-broken_gesture.log
2. while the bug is not present and the gesture is recogniced touch-working_gesture.log

As i said already, the bug can be triggered by emulating the stylus and fixed by using the physical stylus. Hence the bug can be triggered and resolved without triggering anything on the touch device.

emulation vs the real thing

touch

for further narrowing down i looked what "emulating touch" triggers and how it differs from actually touching the device. touch-breaking_gesture_with_emulating_touch.log touch-actually_touching.log -> difference is the additional "(ABS_MT_SLOT), value 0" event in the beginning

stylus

Finally i checked how the emulated stylus and real stylus events look stylus-actual_stylus.log stylus-emulated_stylus_breaks_gestures.log -> difference here is the missing "(BTN_TOOL_PEN), value 0" removal

my best guess

i think the bug is perhaps triggered by xochil waiting for something to finish. as said before triggering a stylus event clears all problems, however triggering a touch event does not. When looking at the bug inducing logs i have the impression there are 2 distinct bugs for each device that both cause xochil to be unhappy.

[pl-semiotics]

@stheid Thank you for the clarifications! Sorry, I didn't realize that you were triggering this via the stylus---since you were discussing gestures, I assumed that "click once on the screen" meant emulated touch input. Thank you for adding on the stylus logs, then---that's certainly necessary.

I think I see the issue. On the RM1 I tested touch mostly by using the menu items since I do use the buttons for navigation, and the menu items continue working fine even when xochitl believes the stylus to be in proximity. The gestures however must be disabled for palm rejection reasons.

So, the problem is that the vnc server sends stylus hover events when you move the mouse cursor around, in case some future application on the tablet will want to take advantage of the stylus position even when it is not touching the display. But, there is no way for it to know when the (virtual) stylus is in/out of proximity, so it ends up thinking the stylus is in proximity forever. When you use the real stylus and then move it away from the display, the out-of-prox event is sent which makes everything happy.

I can think of a few ways to solve it:

• Do not send the stylus-hover-on-cursor-move events at all. This is probably fine for now since xochitl does not use them, but other software might.
• Send an out-of-prox event after a (configurable) timeout after the last cursor movement, and on server shutdown
• Add a VNC protocol extension that we can use to tell the server when the cursor leaves the window, and send out-of-prox on cursor-outside-window, as well as server shutdown/client disconnect from all in-prox clients. Since the client (rmview) is in control, it can choose to send out-of-prox events on a timeout as well (perhaps with a more easily user-configurable timeout). If one wanted to get really fancy, on systems that happen to have their own input from a Wacom tablets attached, one could proxy the computer's tablet's stylus proximity events to the remarkable this way. (If I'm figuring out a way to add VNC extensions anyway, I'd probably support pressure coming from the VNC client as well, rather than the current "it's always max pressure").
• Combination of the previous two: have the server automatically send the event after either a longer timeout or when the cursor leaves the window (or when the server shuts down/the in-prox clients disconnect)

Any thoughts as to which you would prefer @stheid @bordaigorl ? I can do any of them.

[bordaigorl]

Wow thank you so much @stheid for the accurate testing! @pl-semiotics what you are saying makes perfect sense, did not think of the proximity events...

First of all: rmview does not send pointer move events unless a button is pressed; this could simplify a solution because I think in principle it would make sense to think of pointer events as:

1. first button down event -> stylus in proximity + move + pen down events
2. following move events with button down -> move
3. button up event -> move + pen up + stylus out of proximity

For move events with button up state a timeout makes sense (although I don't think they would be useful from rmview and so it's a feature that we wouldn't be using).

An explicit proximity event by extending the protocol could give the highest flexibility I guess (the client is in full control). The easiest way I can think of for doing this is by using a unused button values.

Certainly one should keep track of whether out-of-proximity has been fired so that if can be fired if needed when the server is killed.

[bordaigorl]

Btw I can confirm @pl-semiotics 's diagnosis: the issues I experienced with the RM1 can be reliably triggered by sending stylus or touch events and then resolved by getting the physical stylus close and then far again.

[stheid]

@pl-semiotics nice analysis of the bug, happy to help :+1:

is there any disadvantage to just sending an "out-of-proximity" after each mouse move on the vnc client? i dont understand why waiting for a timeout is useful

I also see the fix for rmview quite easy, but i think it would be cool to solve it once and for all also on the vnc side. I find it really cool to be able to connect to the remarkable with any standard vnc, although i will probably always use rmview as it automatically starts the server on the remarkable. Thats a huge selling point. ;D

[pl-semiotics]

did not think of the proximity events...

Yes, I as well did not think about the lack of out-of-prox. I apologize for the obvious bug!

Btw I can confirm @pl-semiotics 's diagnosis: the issues I experienced with the RM1 can be reliably triggered by sending pointer events and then resolved by getting the physical stylus close and then far again.

Oh, good, I'm glad that we're hopefully only dealing with the one issue here! Thanks for confirming :)

First of all: rmview does not send pointer move events unless a button is pressed; this could simplify a solution because I think in principle it would make sense to think of pointer events as:

1. first button down event -> stylus in proximity + move + pen down events

2. following move events with button down -> move

3. button up event -> move + pen up + stylus out of proximity

For move events with button up state a timeout makes sense (although I don't think they would be useful from rmview and so it's a feature that we wouldn't be using).

Yes, if you are not sending move events with no buttons then it's certainly reasonable to combine the prox events with the button events. I don't know whether or not I'll write something that uses the hover events in the future, but it seems plausible, so I'd like to hang on to the protocol-level ability to send them, though (vs. moving the "only send events while buttons are pressed" into the vnc server).

An explicit proximity event by extending the protocol could give the highest flexibility I guess (the client is in full control). The easiest way I can think of for doing this is by using a unused button values.

Yes, I suppose the only real issue there is what to do with clients that do not support it. As to the specific mechanism to use to convey the information, I shall try to hack something together in the next day or two.

Certainly one should keep track of whether out-of-proximity has been fired so that if can be fired if needed when the server is killed.

Most certainly!

is there any disadvantage to just sending an "out-of-proximity" after each mouse move on the vnc client? i dont understand why waiting for a timeout is useful

Basically, right now it is not significant, but I am generally of the opinion that it's nice to preserve flexibility in case other software comes along later and decides to do something interesting with the stylus-hovering events. I don't know if I will or not, but I might use them in some of my own software in the future as well. Sending in/out events on every mouse motion (that doesn't have buttons pressed) would probably mess with such things, although it is also a reasonable option.

I also see the fix for rmview quite easy, but i think it would be cool to solve it once and for all also on the vnc side. I find it really cool to be able to connect to the remarkable with any standard vnc, although i will probably always use rmview as it automatically starts the server on the remarkable. Thats a huge selling point. ;D

I also use a number of VNC clients, so I'm also concerned about what to do with clients that don't support any particular extension :). However I end up passing the information around, I'll make sure that there's a handshake and that we do something reasonable (like fall back to the in/out-on-every-movement workaround) when client's don't indicate that they want to be smart.

[heikeOnScale]

I wish this feature would be available. I tried to checkout the branch, but it didn't let me. Would it be possible to give me access to this branch?

[bordaigorl]

@heikeOnScale that is weird you should be able to access the branch just fine (it is public). Make sure you fetched it first. Or, you can clone it fresh with git clone -branch pointer-events REPOURL.

[bordaigorl]

@pl-semiotics any news on implementing the proximity events fix? Let me know if I can help.

[stheid]

To be honest, if i where you @bordaigorl i would merge the current version anyways, as the bug is very minor (only effects the gestures) and the feature is large. but obviously it would be nice if it where fixed eventually :D

[heikeOnScale]

@bordaigorl Whatever I try, I always get the same error message: ssh: Could not resolve hostname stheid: Name or service not known As if I wouldn't have the correct access rights. I like to check this branch out, and I am not able to do this.

[bordaigorl]

@heikeOnScale your question seems unrelated to this discussion. If so can you open a new issue and post more details (like your configuration with password omitted). If instead it is on topic, can you clarify how?

[heikeOnScale]

It has been related to my previous comment. I am not able to checkout this branch.

[bordaigorl]

@heikeOnScale ah sorry I forgot about that. Looks like your config isn't right or you are issuing the wrong command. Try cloning the repo with the https URL, or just download the branch from here https://github.com/bordaigorl/rmview/archive/pointer-events.zip

[heikeOnScale]

@bordaigorl I was able to checkout the code, I hadn't realised that the branch name was "pointer-events". And it is such a cool functionality. I love it!

[pl-semiotics]

I think the new test binaries I uploaded for rm2fb/2.6+ support should include the fix for this---if you do want to send prox-but-not-contact events, button 2 can be used.

[bordaigorl]

@pl-semiotics thanks so much for this. I tested the new standalone version on RM1. Everything works flawlessly, but the new prox-but-not-contact event does not solve the issue: I think what you implemented is a way to send "moved closer" events but what we need is "moved away" events. I think that is what the palm rejection is listening for to re-enable the buttons/controls.

[pl-semiotics]

@bordaigorl Strange, I tested this on rM1 as well and I thought that the bug was fixed. The issue before was that there was no way to send out-of-prox events, since every call to submit_wacom_event was passing 1 for the prox parameter. Now it only passes that if button two or button one is held down, so as soon as it receives any event that doesn't have buttons pressed, it ought to work.

Oh, actually, perhaps rmview is only sending events with button 1 down? If so, that would explain why it worked when I tested it but not when you did. If so, could you send an event with no buttons down at the end of the interaction? The component on the tablet needs some way to know when the user stops the click-and-drag interaction. (Sending an out-of-contact+out-of-prox after every received event messes up drawing---we need to only send out-of-contact when the interaction is actually ended).

If that hypothesis is right, you should be able to see that sending a touch event from rmview also restores the touch on the tablet, since receiving any input event without button 1 or 2 down will send the out-of-prox.

[bordaigorl]

@pl-semiotics sorry my fault completely, I messed up a step and I was still using the old executable! It is all working perfectly now! Would you prefer if I waited until you released the new version to include the binaries in rmview?

[pl-semiotics]

@bordaigorl Glad to hear it is actually working!

The only thing I'm waiting on to make a release is confirmation from people who use rm2 that it works (both with xochitl natively and with rm2fb). I don't have any way to test it on rm2 myself, so it's mostly just "well I there was at least one report of it working before the 2.6 upgrade" at the moment.

You may also want to expose some setting for the QSG_RM2FB_PID environment variable, which with the current autodetection code must be set if rm2fb is running not-via-systemd. (If it is running via systemd/toltec, everything ought to work---unless the unit name or the systemd api has been changed.)

[bordaigorl]

You may also want to expose some setting for the QSG_RM2FB_PID environment variable

would that mean finding the pid of rm2fb and invoking the server with QSG_RM2FB_PID=<PID> ./rM2-vnc-server-standalone?

[pl-semiotics]

would that mean finding the pid of rm2fb and invoking the server with QSG_RM2FB_PID=<PID> ./rM2-vnc-server-standalone?

If the rm2fb process is running via systemd as in the packaged versions (which is the easiest way of "finding" it), the backend-qsg will find it for you. If it's running some other way, then yes, that environment variable needs to be set. I don't think it's terribly easy to figure out whether a random fb-owning process is rm2fb or not, so I figured that it'd be easiest to just give a user who wants to run it in their own fashion the ability to specify the pid with via whatever means they want. (I was imagining for rmview perhaps something like a config file setting for a script that returns the pid or some such.)

[bordaigorl]

got it. I think the easiest way would be to allow customizing the path of the server (and disabling automatic upgrade) so that advanced users can use their custom scripts to launch it