Support Unicode Keyboard events in xrdp

[MagicStarTrace]

xrdp: A Remote Desktop Protocol server. Copyright (C) Jay Sorg 2004-2014 See http://www.xrdp.org for more information. Version 0.9.5

In the same environment of Chinese and related dependencies, vnc is a device that can bring external Chinese into the remote ubuntu 18.04 system

I currently use the input method in ubuntu, but I hope that the Chinese from the external client can be brought into the ubuntu system

---

apt install -y language-pack-zh-hans fonts-droid-fallback ttf-wqy-zenhei ttf-wqy-microhei fonts-arphic-ukai fonts-arphic-ukai fonts-arphic-uming

[matt335672]

Hi @huyifanstar

I don't understand what you are trying to do.

Are you trying to get a Chinese keyboard mapping working on the xrdp session? Or are you trying to get the whole locale transferred from the client to the server?

[MagicStarTrace]

Hi @huyifanstar

I don't understand what you are trying to do.

Are you trying to get a Chinese keyboard mapping working on the xrdp session? Or are you trying to get the whole locale transferred from the client to the server?

I hope that the Chinese input from the mac client (rdp client) can be transparently transmitted to the remote instance on the mac through the client

[metalefty]

@matt335672 The feature he wants is called "Unicode input" / "Unicode Keyboard Events" that the client sends inputs as Unicode characters rather than scan codes. See also #441. xrdp partially supports it. It is possible to type ASCII characters in Unicode mode but further uses are not supported because the inputs are transmitted from xrdp to Xorg as keystrokes via virtual keyboard. We need to invent direct passing of Unicode characters to Xorg/Xvnc to implemtnt it.

Android/iOS RDP clients support it. Also, macOS RDP client support it since last year.

It will work like this.

https://user-images.githubusercontent.com/941609/132803539-a07b289f-5bcf-4062-8d0a-7a80ff28d068.mov

[MagicStarTrace]

@matt335672 The feature he wants is called "Unicode input" / "Unicode Keyboard Events" that the client sends inputs as Unicode characters rather than scan codes. See also #441. xrdp partially supports it. It is possible to type ASCII characters in Unicode mode but further uses are not supported because the inputs are transmitted from xrdp to Xorg as keystrokes via virtual keyboard. We need to invent direct passing of Unicode characters to Xorg/Xvnc to implemtnt it.

Android/iOS RDP clients support it. Also, macOS RDP client support it since last year.

It will work like this.

RPReplay_Final1631250814.mov

Convert Unicode characters to Xorg/Xvnc , Is there any way I can configure it to enable support? Configuration file ?

Thank You

[metalefty]

t is possible to type ASCII characters in Unicode mode but further uses are not supported because the inputs are transmitted from xrdp to Xorg as keystrokes via virtual keyboard. We need to invent direct passing of Unicode characters to Xorg/Xvnc to implemtnt it.

You didn't read here.

[matt335672]

Thanks for the explanation.

@huyifanstar - I'm going to change the title of this issue to reflect @metalefty's comments above.

[MagicStarTrace]

@matt335672 The feature he wants is called "Unicode input" / "Unicode Keyboard Events" that the client sends inputs as Unicode characters rather than scan codes. See also #441. xrdp partially supports it. It is possible to type ASCII characters in Unicode mode but further uses are not supported because the inputs are transmitted from xrdp to Xorg as keystrokes via virtual keyboard. We need to invent direct passing of Unicode characters to Xorg/Xvnc to implemtnt it.

Android/iOS RDP clients support it. Also, macOS RDP client support it since last year.

It will work like this.

RPReplay_Final1631250814.mov

Adjusted to Unicode on mac, just tried it, it did not meet expectations Is there any progress in this area?

Eventually the xrdp server will support Chinese Unicode keyboard events?

Thank you!

https://user-images.githubusercontent.com/15685376/132838112-d4c1b67b-7ed4-4ec1-a601-4c1965e96421.mov

[zsj1029]

Run these commands to generate the keyboard mapping for your chosen input. 00000804 is chinese keyboard Mapping Code xrdp-genkeymap km-00000804.ini To copy it in the xrdp folder

sudo mv km-00000804.ini /etc/xrdp Remember that you must change the permissions of the file, so that it can be used

sudo chown root:root /etc/xrdp/km-00000804.ini Restart the service and it should work

sudo service xrdp restart

That'all . Enjoy it

[zsj1029]

Maybe Xrdp could set km-00000804.ini as default installation

[MagicStarTrace]

Maybe Xrdp could set km-00000804.ini as default installation Ok, thanks a lot!

Where can I get this (km-00000804.ini)?

Do I need to make special settings on the client side?

Thank You!

[zsj1029]

Run these commands to generate the keyboard mapping for your chosen input. 00000804 is chinese keyboard Mapping Code xrdp-genkeymap km-00000804.ini To copy it in the xrdp folder

sudo mv km-00000804.ini /etc/xrdp Remember that you must change the permissions of the file, so that it can be used

sudo chown root:root /etc/xrdp/km-00000804.ini Restart the service and it should work

sudo service xrdp restart

That'all . Enjoy it

Just run command upon Client no need make special settings Change to chinese input method, it works

[seflerZ]

Run these commands to generate the keyboard mapping for your chosen input. 00000804 is chinese keyboard Mapping Code xrdp-genkeymap km-00000804.ini To copy it in the xrdp folder sudo mv km-00000804.ini /etc/xrdp Remember that you must change the permissions of the file, so that it can be used sudo chown root:root /etc/xrdp/km-00000804.ini Restart the service and it should work sudo service xrdp restart That'all . Enjoy it

Just run command upon Client no need make special settings Change to chinese input method, it works

Tried but doesn't work. @zsj1029 Did I miss something?

[zsj1029]

Run these commands to generate the keyboard mapping for your chosen input. 00000804 is chinese keyboard Mapping Code xrdp-genkeymap km-00000804.ini To copy it in the xrdp folder sudo mv km-00000804.ini /etc/xrdp Remember that you must change the permissions of the file, so that it can be used sudo chown root:root /etc/xrdp/km-00000804.ini Restart the service and it should work sudo service xrdp restart That'all . Enjoy it

Just run command upon Client no need make special settings Change to chinese input method, it works

Tried but doesn't work. @zsj1029 Did I miss something?

make sure u have installed Chinese input method Fcitx in your OS first

[seflerZ]

@zsj1029 Here is the log. As you can see all keyboard maps has been found. I'm using Fcitx to use WuBi-86 input method too.

Would you mind to provide the logs you have successfully enabled?

[MagicStarTrace]

Run these commands to generate the keyboard mapping for your chosen input. 00000804 is chinese keyboard Mapping Code xrdp-genkeymap km-00000804.ini To copy it in the xrdp folder sudo mv km-00000804.ini /etc/xrdp Remember that you must change the permissions of the file, so that it can be used sudo chown root:root /etc/xrdp/km-00000804.ini Restart the service and it should work sudo service xrdp restart That'all . Enjoy it

Just run command upon Client no need make special settings Change to chinese input method, it works

root@S0-GPU:/etc/xrdp#xrdp-genkeymap km-00000804.ini xrdp-genkeymap: unable to open display ''

How should this be resolved?

[seflerZ]

Well, I digged into the code recently. You can't send unicode to remote directly at present because xrdp simply doesn't support it. It does accept unicode input at method "xrdp_wm_key_unicode()" in file "xrdp_wm.c". But it converts it into scan code by comparing the glyphs and invokes "xrdp_wm_key()".

The reason xrdp module send scan code is because in the backend xorgxrdp, it simulated a virtual keyboard and uses "xf86PostKeyboardEvent()" to send keys. This function supports scan codes only.

I think what @zsj1029 said it can support other keyboard layouts other than en-US. For example, you can type in Greece glyphs or Japanese hiragana with unicode input enabled. It might work because it can be mapped to keys on the board. But you can not input other unicode which is not mapped to keys. For example, the Chinese has more than 10k characters, it's impossible to map them to the keyboard directly. Therfore, a way must be found to support input unicode directly, other than simulating keypresses.

As what I said, the "xf86PostKeyboardEvent()" supports scan code only. I don't know which X Window API supports it so I can upgrade the xorgxrdp input driver. @metalefty do you have any idea?

[matt335672]

I'm not as familiar with X11 internals as some of the other developers here, but I suspect the keycode model is too deeply embedded in X11 to implement this, at least with X11 backends.

The RFB protocol used by the VNC back end is documented here:-

https://github.com/rfbproto/rfbproto/blob/master/rfbproto.rst

There's no provision for providing generic Unicode input as keypresses.

X11 key events (which are visible to the application) are here. They are keycode dependent:-

https://tronche.com/gui/x/xlib/events/keyboard-pointer/keyboard-pointer.html

Despite the 'unsigned int' in the above, on the wire, X11 keycodes are only 8 bits, which gives us very limited possibilities:-

https://x.org/releases/X11R7.7/doc/xproto/x11protocol.html

I'll be very happy to be proved wrong here though, if anyone has better info than I have.

[seflerZ]

@matt335672 Okay, thank you. It seems X Server API takes over of hardware abstration only. We can make another way, to take advantage of iBus or Fcitx input method APIs to generate characters. It is centain that we can implement our iBus or Fcitx engine and accepts requests from xorgxrdp to input unicode characters through IPC calls. But this approch is difficult, costly and hard to maintain. I hope there is a way to send requests to iBus directly to generate unicode characters. I'm working on it.

[matt335672]

That sounds doable, @seflerZ but I agree with all of "difficult, costly and hard to maintain".

At the moment, the RDP input PDUs are send directly from xrdp to xorgxrdp or VNC. It might be possible to send the Unicode PDUs to chansrv instead and get that to interface to the session iBus. That's fine providing chansrv is connected to xrdp. There are still a few bugs in this area as I'm sure you are aware.

[seflerZ]

Hi, @matt335672 . I found it's much easier than I expected. We can implement a tiny iBus input engine within 100 code lines and use these functions to send unicode characters to iBus.

// Unicode identifier for Chinese character "你"
gunichar chr = 20320;
ibus_engine_commit_text(engine, ibus_text_new_from_unichar(chr));

As for the engine, we can use "ibus_engine_desc_new()" and "ibus_component_add_engine()" APIs to create a temporary one.

So the question becomes, where shall we put this implementation, in "xrdp" or "xorgxrdp" module? I think in "xrdp" is more approachable, since "xorgxrdp" is designed for dealing with X Window system.

[matt335672]

It really depends on how the IPC works for ibus.

ibus runs on top of glib2, so it probably uses D-Bus, although I'm not sure about this.

Neither xrdp or xorgxrdp have a connection to the session D-Bus. I think you'll need to get the data into xrdp-chansrv which has full access to the session.

That's going to be quite fiddly. By all means have a go, but if you'd like a hand let me know what branch of xrdp you're working on.

[jsorg71]

I think @matt335672 is right, if you want to talk to ibus, chansrv is the place to do it. There is already some non channel message in chansrv. I think the rail code can send RDP orders for new window or update window for example.

I don't think ibus is good for general input. I'll suggest another possibility. If we can map unicode to keysyms we could do something like xdotool. It looks if the keysym is in the current map, if it is, it uses it, if not, it looks for the first free scancode and maps the new keysym to the scancode, uses it, then removes it. You can use xev and xdotool to see how it works. We could do this in xorgxrdp where we already have a communication path.

[seflerZ]

I agree that iBus is not a general input solution. But if we implement our own input system, we'll need to deal with various window frameworks like Qt\Gtk2.0-4.0\x11\Wayland etcetera. That will be a big project...

As for "keysyms", It does map unicode characters to keys on the keyboard. But as I memtioned, some language like Chinese has more than 10k characters (Japanese also use characters origin from Chinese), we must simulate a gaint keyboard with thousands of keys to hold all the unicode characters. It is doable but some what weird though. (Besides, xrdp limited the size of keys to 256 at present.)

Of course, it will be great if we can find any simple and elegant approach.

[akallabeth]

would be really great to have this implemented, modern clients more and more start to use this as they do not have a keyboard to forward but some touch interface. (using UnicodeKeyboard is a huge help there as the guessing game which keyboard layout is active can be avoided)

[matt335672]

I can't see a simple and elegant approach myself I'm afraid:-

1. ibus isn't by any means globally used.
2. The reconnection to chansrv is quite flaky at times for current xrdp versions. We'd have to solve this for a robust solution based on chansrv.
3. Jay's method is (I think) going to generate a lot of MappingNotify events for all X applications. We also can't use it for the few remaining use-cases where a VNC backend is necessary.

If we did implement something in chansrv, a production solution would need a way for the user to configure the desktop input method - a global method wouldn't be adequate.

At the same time, if @seflerZ want to try modifying chansrv for prototyping purposes I'm happy to give him a hand. We'll learn more that way about whether a robust solution can be found.

[matt335672]

@seflerZ

I had a think about this over the weekend, and had a look at the code myself. It's pretty hard to find your way around it, so I thought I'd put something together quickly to get you started. That way you can focus on the immediate problem rather than worrying about how xrdp is put together.

I've implemented a basic method to send unicode input from xrdp to chansrv. Code is here, based on the devel stream:-

https://github.com/matt335672/xrdp/tree/initial_ibus_testing

This code adds an --enable-ibus switch to configure. If this is specified, Unicode keyboard events are passed to chansrv (if it's up) and simply logged in chansrv in process_message_unicode_key_press().

I've added the relevant libraries to chansrv, so you should be able to expand the code in there and call ibus methods.

This is pretty basic and would need more work before it went into production. After you've played with it, we can make it more robust if you think between us we can make something useful.

Also, being based in the UK, I can't find a way to test this here. If you have any suggestions, please let me know.

[Kishi85]

Also, being based in the UK, I can't find a way to test this here. If you have any suggestions, please let me know.

Testing should be easily done by using a phone/tablet and trying to send emojis from the on-screen keyboard as they are sent as Unicode characters via Microsofts Android RDP client

[akallabeth]

Also, being based in the UK, I can't find a way to test this here. If you have any suggestions, please let me know.

Testing should be easily done by using a phone/tablet and trying to send emojis from the on-screen keyboard as they are sent as Unicode characters via Microsofts Android RDP client

xfreerdp3 supports reading symbols from a pipe (check the command line help) that way you can just echo utf-8 encoded stuff to a pipe and it is sent as ucs-2 encoded data.

[matt335672]

Thanks both - that's very useful indeed.

@akallabeth - I spent a bit of time looking about in freerdp3 yesterday and completely missed the pipe option! For the record, I'm now using a command line like the following :-

xfreerdp /gfx /kbd:pipe:/tmp/input.pipe /v:<hostname> /size:1280x1024 /u:testuser /d: /p:<password>

I can then push stuff through xrdp as expected:-

echo -n '😉' >/tmp/input.pipe

@seflerZ - I've fixed a significant bug in the code I pushed. Also, I've remapped surrogate pairs as well, so I can send smileys in. I'm getting log messages like the following in chansrv:-

2024-03-26T10:12:41.980+0000] [INFO ] [process_message_unicode_key_press(chansrv.c:844)] Unicode char 0x00000073 received down=1
[2024-03-26T10:12:41.175+0000] [INFO ] [process_message_unicode_key_press(chansrv.c:844)] Unicode char 0x00000073 received down=0
[2024-03-26T10:12:43.443+0000] [INFO ] [process_message_unicode_key_press(chansrv.c:844)] Unicode char 0x0000006c received down=1
[2024-03-26T10:12:43.613+0000] [INFO ] [process_message_unicode_key_press(chansrv.c:844)] Unicode char 0x0000006c received down=0
[2024-03-26T10:12:43.638+0000] [INFO ] [process_message_unicode_key_press(chansrv.c:844)] Unicode char 0x0000006c received down=1
[2024-03-26T10:12:43.715+0000] [INFO ] [process_message_unicode_key_press(chansrv.c:844)] Unicode char 0x0000006c received down=0
[2024-03-26T10:12:43.917+0000] [INFO ] [process_message_unicode_key_press(chansrv.c:844)] Unicode char 0x00000073 received down=1
[2024-03-26T10:12:43.966+0000] [INFO ] [process_message_unicode_key_press(chansrv.c:844)] Unicode char 0x00000073 received down=0
[2024-03-26T10:12:44.719+0000] [INFO ] [process_message_unicode_key_press(chansrv.c:844)] Unicode char 0x00000073 received down=1
[2024-03-26T10:12:44.139+0000] [INFO ] [process_message_unicode_key_press(chansrv.c:844)] Unicode char 0x00000073 received down=0
[2024-03-26T10:12:58.887+0000] [INFO ] [process_message_unicode_key_press(chansrv.c:844)] Unicode char 0x0001f602 received down=1
[2024-03-26T10:12:59.106+0000] [INFO ] [process_message_unicode_key_press(chansrv.c:844)] Unicode char 0x0001f602 received down=0
[2024-03-26T10:13:36.665+0000] [INFO ] [process_message_unicode_key_press(chansrv.c:844)] Unicode char 0x0001f609 received down=1
[2024-03-26T10:13:36.737+0000] [INFO ] [process_message_unicode_key_press(chansrv.c:844)] Unicode char 0x0001f609 received down=0

[seflerZ]

@matt335672 Thanks. I'm dealing with the iBus now. Got some progresses but also problems. But I think it won't take too much long.