Implement keyboard event handling

[bk138]

Implement full keyboard handling so that a remote viewer can send keyboad events and does not have to rely on the Android soft keyboard.

If you'd like to sponsor implementing this feature, you can do so at https://issuehunt.io/r/bk138/droidVNC-NG/issues/4

[meiskam]

fyi alpha vnc lite supports keyboard events

looks like they created their own "keyboard", and routed keypresses through that https://developer.android.com/guide/topics/text/creating-input-method

[sudomain]

I might give this a try but have an implementation question:

... does not have to rely on the Android soft keyboard.

Is there a (non-root) way to send key events in Android without changing the soft keyboard? I know of apps, such as KeyMapper and KeyPassDX, which can input text for an user, but they do via their own soft keyboard implementations.

[bk138]

I think this is called input method extension aka IME.

[meiskam]

creating a keyboard/IME and routing keys through it doesn't require changing the user's soft keyboard

[meiskam]

this could also be implemented with adb shell cmd input

root@barbet:/system/bin # cmd input
Usage: input [<source>] [-d DISPLAY_ID] <command> [<arg>...]

The sources are:
      dpad
      keyboard
      mouse
      touchpad
      gamepad
      touchnavigation
      joystick
      touchscreen
      stylus
      trackball

-d: specify the display ID.
      (Default: -1 for key event, 0 for motion event if not specified.)
The commands and default sources are:
      text <string> (Default: touchscreen)
      keyevent [--longpress|--doubletap] <key code number or name> ... (Default: keyboard)
      tap <x> <y> (Default: touchscreen)
      swipe <x1> <y1> <x2> <y2> [duration(ms)] (Default: touchscreen)
      draganddrop <x1> <y1> <x2> <y2> [duration(ms)] (Default: touchscreen)
      press (Default: trackball)
      roll <dx> <dy> (Default: trackball)
      motionevent <DOWN|UP|MOVE|CANCEL> <x> <y> (Default: touchscreen)
      keycombination <key code 1> <key code 2> ... (Default: keyboard)

[ghost]

wifikeyboard (F-Droid) is a simple example

[Holzhauer]

wifikeyboard does not work for me an Android 12 (website available, but "Status: Connection problem"). Seems out of date.

[ghost]

wifikeyboard does not work for me an Android 12 (website available, but "Status: Connection problem"). Seems out of date. wifikeyboard on Android 9 Have you set it as the default IME? https://user-images.githubusercontent.com/107785860/174596382-dedb7d70-31f4-4a4d-a7fb-b7d06c103c6a.mp4

[sudomain]

I guess wifikeyboard is technically not unmaintained, but its last real update (not a translation or typo fix) was in 2013. If other solutions are being considered, consider scrcpy

Edit: There's also ADBKeyboard which I've used, but this requires the user to know how to send intents

[bigrck64]

Having keyboard support is a feature I would love to see in droidVNC-NG, virtual keyboard seems to be the standard in other vnc server.

[arminlinder]

Cudos to bk136, the software does work very well on my mobile and my tablet, and, working as a teacher, it gives me the opportunity to show my pupils what I can do with my Android devices via a beamer.

I do strongly assist that remote keyboard support be implemented, as well as true two-way clipboard support.

I do also have ways to spend some real money on the feature, the school has a certain budget I could spend. Unfortunately, AFAIK, the funding campaign above has not raised any funds yet, and I'd hate to sink money into a feature which then does never get implemented because funds are not high enough.

Dear bk136, what is the amount of funds you need to see, before you start working on implementing these two features?

[bigrck64]

Good question to bk136, I can also pays some funds to have keyboard support :) So what is your money target for that ?

[bk138]

Hi @arminlinder @bigrck64 my usual rate for professional work is 90€/h; this feature might take 3 or more work days to complete. That said, money isn't the blocker here, but time. I'd actually like to work on this in my free time, but currently don't have much :-/. The rough plan at the moment is to finish #98 which will allow closing of a bunch of other issues, then tackle this one here. Nonetheless, some reasonable funding in https://issuehunt.io/r/bk138/droidVNC-NG/issues/4 might be a nice incentive (for me or anyone else) 😉.

[bigrck64]

Thanks for coming back to us bk138 :) On my side I can give $300, but issuehunt.io only accept credit card payment, and I would like to use Paypal for this.

[bk138]

@bigrck64 thanks for the commitment! Let's do it the mutual trust way: you keep the investment until the work is done, afterwards you can wire it to the implementer (me or maybe someone else who's faster). What do you think?

[bigrck64]

Good idea! I will keep this money and send it to you (or the faster one ;-) when the work will be done. @bk138: If it's you, could you accept Paypal payment ?

[bk138]

Good idea! I will keep this money and send it to you (or the faster one ;-) when the work will be done. @bk138: If it's you, could you accept Paypal payment ?

Yes.

[bigrck64]

Yes

Perfect :)

@arminlinder: will you be able to join me and help to found this dev ?

[Haiz14]

Which branch is this feature being worked on maybe I can contribute?

[bk138]

Which branch is this feature being worked on maybe I can contribute?

@Haiz14 no branch at all yet. What's needed first is some research on the best approach, some are listed here. What's especially interesting is how others do it (and what issues they might be facing). Happy to have a google doc or other means of shared doc-keeping!

[Haiz14]

Tasker can do this after being granted android.permission.WRITE_SECURE_SETTINGS

This is because this permission allows the app to modify system settings,
 including input-related settings.

To simulate keyboard text input, the app can use the "input"
 command provided by the Android Debug Bridge (ADB) tool, 
which allows for simulating touch and keyboard input events. 
The app can send input events to the system, 
which will be interpreted as if the user had physically 
typed the corresponding keys on the keyboard.

[swilson2022]

Hi, great work on the app so far, extremely useful so thank you for that.

I was just wondering if there are any updates on the implementation of keyboard handling?

It is way beyond my capabilities to offer any help in the coding, however I would be willing to sponsor to help push it across the finish line as soon as possible.

Thanks in advance.

[bk138]

@swilson2022 Investing some time into research on how others do it would help. Alternatively, see https://github.com/bk138/droidVNC-NG/issues/4#issuecomment-1343301749

[luweitest]

This is the main reason I'd like to use VNC: to use computer input method. I've tried "alpha vnc lite" but it does not support computer input either (through plain keyboard input could work, which droidvnc-ng does not support). Hope there's some good news soon.

[carylewis]

I am very interested in this feature as well.

Alternatively, has anyone thought of tackling this from the client end, and automating the control of the soft keyboard?

[luweitest]

On 2023-8-15 22:17, Cary Lewis wrote:

I am very interested in this feature as well.

Alternatively, has anyone thought of tackling this from the client end, and automating the control of the soft keyboard?

Maybe analyzing how "alpha vnc lite" handles this could shed some light, though it only supports bare English keyboard, no other characters by IME.

-- Regards, Lu Wei IM: @.*** PGP: 0xA12FEF7592CCE1EA

[bk138]

@luweitest go ahead if you have time and post your findings here - any help appreciated :heart: !

[luweitest]

On 2023-8-16 15:27, Christian Beier wrote:

@luweitest https://github.com/luweitest go ahead if you have time and post your findings here - any help appreciated ❤️ !

I get a recommendation at group comp.mobile.android that scrcpy works: https://github.com/Genymobile/scrcpy . I tested and it does work with all IME. Although it's not a VNC solution but maybe you could get some hint from it. Good luck!

-- Regards, Lu Wei IM: @.*** PGP: 0xA12FEF7592CCE1EA

[macamba]

Hi all,

I was doing research about how VNC server get send events to Android no root

It only use AccessibilityService to get that. This implement can send any global event only over field text, for example when the focus is in contact phones over name contact edition, so only in text field free but it will not work if you try to send these event over Desktop, you can try yourself sending D-Pad directions by VNC viewer to server remote to move between icon applications or what ever, it will not work. To get that it uses "findFocus" to get field text and work over it

To get send few global keyevents it uses function performGlobalAction(int value) Home, Back, PowerDialog, Applications Opened, Capture Screen, Off Screen/Standby, and some actions more performGlobalAction(1) performGlobalAction(2) ...

Also this implement let send click mouse, not right and middle button mouse, I see something quick in my first search in google about mouse event so it shouldn't be difficult find something about that. To do that it uses "DispatchGesture"

[IssueHuntBot]

@panicride has funded $50.00 to this issue.

---

• Submit pull request via IssueHunt to receive this reward.
• Want to contribute? Chip in to this issue via IssueHunt.
• Checkout the IssueHunt Issue Explorer to see more funded issues.
• Need help from developers? Add your repository on IssueHunt to raise funds.