Using ADBKeyboard for injecting text

[excitoon]

[Artur202030]

https://github.com/Lurker00/scrcpy/releases/tag/v1.10L0 The new features are: -10-fingers multitouch support. -Any language input support with an aid of ADBKeyBoard (included). ...

[rom1v]

Hi,

I just implemented what I suggested here, and now it works great (without latency).

You could test my work-in-progress branch adbkeyboard.

scrcpy --use-adb-keyboard

What remains to do:

• automatically select the ADB keyboard on start
• automatically restore the initial keyboard on stop (even if that won't work if we start 2 instances of scrcpy with ADBKeyboard, since the second one will consider that the initial keyboard was ADBKeyboard).
• some cleanup

[excitoon]

Hey.

You can just never restore keyboard to ADBKeyboard, that shall solve the problem.

Kind regards, Vladimir.

On Mon, Nov 9, 2020, 11:47 PM Romain Vimont [email protected] wrote:

Hi,

I just implemented what I suggested here https://github.com/Genymobile/scrcpy/pull/1751#discussion_r486393847, and now it works great (without latency).

You could test my work-in-progress branch adbkeyboard https://github.com/Genymobile/scrcpy/commits/adbkeyboard.

scrcpy --use-adb-keyboard

What remains to do:

• automatically select the ADB keyboard on start
• automatically restore the initial keyboard on stop (even if that won't work if we start 2 instances of scrcpy with ADBKeyboard, since the second one will consider that the initial keyboard was ADBKeyboard).
• some cleanup

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Genymobile/scrcpy/pull/1751#issuecomment-724268986, or unsubscribe https://github.com/notifications/unsubscribe-auth/AARGSB6AQO3LGR3TBX7DUDTSPBIO3ANCNFSM4RFC3LEQ .

[rom1v]

A future next step could be to include the IME in scrcpy (as a real android app), to avoid the requirement to install a separate app.

I'm wondering if it's still possible to create a real/installable android app without gradle (using build-tools directly). The reason is that the official debian package uses build_without_gradle.sh because the full Android SDK is not available in Debian official repositories.

[Artur202030]

hi, both English and Russian letters are printed simultaneously.

[rom1v]

@Artur202030 Try with:

scrcpy --use-adb-keyboard --prefer-text

[rom1v]

An issue with using the existing ADBKeyBoard is the absence of permissions.

As a consequence, once enabled, any installed app could inject any text or events (anything managed by AdbIME) as they like. This is a security issue IMO.

To avoid the problem, the IME should declare a permission, and with shell rights we can send a broadcast with this permission. In short, the equivalent of (not tested):

adb shell am broadcast -a com.com.myexample.MY_ACTION --receiver-permission com.myexample.permission.MY_PERMISSION

Note that this may only work for broadcasts, not services or activities. For these cases (or if the solution above does not work), we could require some shell permission. But it will cause issues for XiaoMI devices: https://github.com/Genymobile/gnirehtet/issues/5 (and gnirehtet may not be started by broadcast receivers https://github.com/Genymobile/gnirehtet/commit/f8681809a28a0073f791f0d9eae65da02ae73c3d).

[Artur202030]

@ Artur202030 Попробуйте:

scrcpy --use-adb-keyboard --prefer-text

Sorry your code works without delay

[rom1v]

so normal but 1 sec delay

Which branch?

[Artur202030]

i tested your code and excitoon. Mixed up files. Sorry.

[rom1v]

Could you confirm that there is no delay on my branch (https://github.com/Genymobile/scrcpy/pull/1751#issuecomment-724268986)?

[Artur202030]

Could you confirm that there is no delay on my branch (#1751 (comment))?

Your code works without delay. Thanks.

[quyleanh]

@excitoon could you please take a look at this PR again?

[quyleanh]

@rom1v I have built my self adbkeyboard and used scrcpy --use-adb-keyboard, but I still cannot enter Vietnamese character.

Is there any way to report log?

[quyleanh]

@rom1v could you please take a look at this PR?

[rom1v]

@quyleanh I implemented broadcasts so that it works without latency (https://github.com/Genymobile/scrcpy/pull/1751#issuecomment-724268986), but the absence of permissions of AdbKeyboard is a problem (https://github.com/Genymobile/scrcpy/pull/1751#issuecomment-724532949). Also, scrcpy should enable/disable the keyboard on start/stop.

I consider adding a feature to "install" the scrcpy server instead. That way, it could provide its own IME: https://github.com/Genymobile/scrcpy/issues/1880#issuecomment-725655830

but I still cannot enter Vietnamese character.

It should allow to inject any UTF-8 chars (try to copy paste a vietnamese character with Alt+Shift+v to inject characters).

[quyleanh]

@rom1v thank you for your kind response.

It should allow to inject any UTF-8 chars (try to copy paste a vietnamese character with Alt+Shift+v to inject characters).

I know that copy paste feature. But I would like to inject UTF-8 characters directly from keyboard.

@quyleanh I implemented broadcasts so that it works without latency (#1751 (comment)), but the absence of permissions of AdbKeyboard is a problem (#1751 (comment)). Also, scrcpy should enable/disable the keyboard on start/stop.

I consider adding a feature to "install" the scrcpy server instead. That way, it could provide its own IME: #1880 (comment)

So in summary, we will not have inject UTF-8 character directly from keyboard soon? As I understand, there are a lot of problem needed to do.

[rom1v]

I know that copy paste feature. But I would like to inject UTF-8 characters directly from keyboard.

Yes, I know, but since you said that Vietnamese characters were not injected, I wanted to check where it failed.

There are two copy-paste:

• once using the clipboard, where any byte stream work (including UTF-8)
• once injecting the characters, which acts as if every character was pressed from a keyboard (injected char by char), currently it only supports ASCII (+ some other accented chars)

If you use the second method with the ADBKeyboard, I expect that Vietnamese characters work. Is it the case?

[quyleanh]

If you use the second method with the ADBKeyboard, I expect that Vietnamese characters work. Is it the case?

Yes, I use the second method. There are only some Vietnamese characters work. For example, if I type ô (unicode hex is U+00F4), it is fine. But if I type ồ (unicode hex is U+1ED3), the warning appreases. [server] WARN: Could not inject char u+1ed3

You can refer to this summary table.

[rom1v]

[server] WARN: Could not inject char u+1ed3

You're not running my branch adbkeyboard with scrcpy --use-adb-keyboard: https://github.com/Genymobile/scrcpy/pull/1751#issuecomment-724268986

[quyleanh]

You're not running my branch adbkeyboard with scrcpy --use-adb-keyboard: #1751 (comment)

I build your branch and run it now, but I forgot to use scrcpy --use-adb-keyboard syntax. I checked again with scrcpy --use-adb-keyboard, but I cannot enter any Vietnamese characters now. How can I provide log since there is no output or warning on terminal?

[rom1v]

Try with AdbKeyboard alone: https://github.com/senzhk/ADBKeyBoard (without scrcpy).

[quyleanh]

Thank you for your suggestion but I want to use scrcpy for both mirroring screen and injecting text. There is no way after all?

[rom1v]

Yes, I got it, but try with AdbKeyboard alone to check if Vietnamese characters work. If they don't, then it's expected that the adbkeyboard branch can't do it either. If it works, then there is a problem somewhere else.

[quyleanh]

Thank you for your detail instruction. The senzhk/ADBKeyBoard works well. This is my test:

adb shell am broadcast -a ADB_INPUT_TEXT --es msg 'Hôm nay trời rất là đẹp. Ồ ngoài kia có chú ngựa ô. Ổ gà. Ở nhà không đi học. Hỏi han. Hello?'
Broadcasting: Intent { act=ADB_INPUT_TEXT flg=0x400000 (has extras) }
Broadcast completed: result=0

Is there any other suggestion?

[rom1v]

Test with ints:

adb shell am broadcast -a ADB_INPUT_CHARS --eia chars 7891

(7891 is 0x1ed3)

[quyleanh]

It's fine. The result is ồ.

[rom1v]

OK so in theory it should work with scrcpy on adbkeyboard. I don't know why it does not, but will not investigate for now, since in the end scrcpy should have its own IME (https://github.com/Genymobile/scrcpy/pull/1751#issuecomment-744586560).

[quyleanh]

Actually I have to use an external IME to type and inject Vietnamese on Windows. However other apps are OK with it. In the previous test, all Vietnamese characters are injected from terminal since I already paste it in adb command. I don't know is there any different between injecting directly from adb command and injecting from other IME.

By the way, I'm glad to hear that you are working on a work around. Could you please push it in this repo, so I can try it? Or if it needs to test, just let me know.

[Helaer]

@rom1v Using ADBKeyboard to input text is a good solution, which can solve the problem that Samsung and other devices cannot paste text, Why is there no progress?