Unicode input for Linux doesn't work for all applications

[LukeXuan]

Depending on the application, CTRL and SHIFT might need to be pressed for the whole sequence (except for the terminating SPACE). Here's the script I'm using right now,

ZMK_BEHAVIOR(iris_sep, macro,
    wait-ms = <5>;
    tap-ms = <5>;
    bindings = <&macro_press &kp LCTRL &kp LSHFT> ,
               <&macro_tap &kp U &kp N2 &kp N2 &kp N1 &kp N7 &kp SPACE> ,
               <&macro_release &kp LSHFT &kp LCTRL>;
)

For my limited test (well, Firefox, thunderbird, and Emacs), "pressing throughout" style works for all of them, but the current macro doesn't work in Emacs.

[nicenemo]

You might get similar issues on Windows. e.g. in notepad it does work but in Powershell it does not. In GMail or Google Chat it will not work either. Alt least when manually typing a 4 digit number and thenalt shift X. I don't think these macros can fix that.

[SammyJenas]

Not sure if you're still having this issue, but the following works for me in Emacs and Firefox:

ZMK_BEHAVIOR(em_dash, macro,
    wait-ms = <5>;
    tap-ms = <5>;
    bindings = <&macro_press &kp LCTRL &kp LSHFT> ,
               <&macro_tap &kp U &kp N2 &kp N0 &kp N1 &kp N4 &kp SPACE> ,
               <&macro_release &kp LSHFT &kp LCTRL>;
)

Here's my full config.

--EDIT-- Might have misunderstood your issue. If you were referring to the ZMK_UNICODE helper, then I had the same issue, i.e. the following works in Firefox but not Emacs:

ZMK_UNICODE_SINGLE(em_dash, N2, N0, N1, N4)

[LukeXuan]

Yes, I was proposing to change the ZMK_UNICODE helper to the one I suggested (which is the same way you are doing right now).

[SammyJenas]

I see. While this way of doing it works for me in Emacs, Firefox, and LibreOffice, it doesn't work in the terminal (Foot/Kitty) or in the Brave browser...

[urob]

FYI, the activation sequence for the Unicode helpers can be customized using OS_UNICODE_LEAD and OS_UNICODE_TRAIL.

For instance, to hold both LCTRL and LSHFT while the actual unicode sequence is sent, one can set:

// Note: These must be set _before_ including helper.h.
#define OS_UNICODE_LEAD &macro_press &kp LCTRL &kp LSHFT &macro_tap &kp U
#define OS_UNICODE_TRAIL &macro_tap &kp SPACE &macro_release &kp LCTRL &kp LSHFT

#include "zmk-helpers/helper.h"

// ...

Using HOST_OS is just a shortcut to set OS_UNICODE_LEAD and OS_UNICODE_TRAIL to the same default values that are used by QMK.

I'm open to changing the Linux default if the QMK default doesn't work for the majority of people. But as this would be a breaking change, I would like to make sure that the current default indeed isn't working for the majority of Linux users. In the meantime, please let me know if there are any issues with overwriting the defaults using OS_UNICODE_LEAD/TRAIL.

[Roverp1]

I'm also have issues using a unicode-based russian layer on linux. The configuration works, but the input is unusable in most apps.

• In Telegram, I get raw Unicode sequences like 0439 0446 0443 043a instead of Cyrillic letters.
• In Firefox, I can see the Unicode sequence being typed out before it transforms into the correct character.
• In the kitty terminal, Unicode input doesn't work at all.

I did include the necessary headers in my config:

#define HOST_OS 1
#include "zmk-helpers/helper.h"
#include "zmk-helpers/unicode-chars/russian.dtsi"

So it looks like the problem is not with the config

[LukeXuan]

I don't think there is a universal solution on Linux. I suggest changing the README file to make the following points clear:

• different applications might expect different input methods for unicode sequences, make sure to test it before configuration,
• document the usage of OS_UNICODE_LEAD and OS_UNICODE_TRAIL, and maybe put the examples in this issue into README as well.

[urob]

Added additional documentation in f8d5ac771aed291ea54215bf1b6bc6f67336477d for how to customize input on Linux.

[urob]

I'm also have issues using a unicode-based russian layer on linux. The configuration works, but the input is unusable in most apps.

* In Telegram, I get raw Unicode sequences like `0439 0446 0443 043a` instead of Cyrillic letters.

This should probably be discussed in a separate issue if you want to open one. I have seen issues like this before due to the default timings being too small on some system/keyboard/bluetooth-strength combinations. We should probably make this configurable.

[urob]

For anyone following this, it might be worth checking out the new [zmk-unicode](https://github.com/urob/zmk-unicode) module.

There's two improvements related to the issues raised in this thread:

1. There's now two Linux input methods, which can be switched while the keyboard is in use. Reading up a bit more on Unicode input in Linux, those two methods should cover most applications. They are also customizable to accommodate other needs.
2. The implementation is much more robust under the hood which should help with a race condition the old implementation suffered from (*)

---

(*) Notes

The old implementation using only preprocessor macros was limited by how behavior queues were implemented by ZMK's macro and mod-morph behaviors. The issue was that in order for Unicode input to work, one must exactly control which mods are active during the input process: That is, one wants to mask everything that might be pressed by the user (including shift), and one wants to selectively unmask what's required for the input.

The issue with this is that mod-morph would immediately unmask all mods right after the macro queued all its keys. For there to be any hope of success, we had to set both the tap and wait times to zero. Even then, there was a race condition, which on slow connections would result in garbled inputs.

The new implementation fixes this by (1) handling the masking / unmasking much more carefully, (2) queuing the unmasking until the entire sequence is finished, (3) increasing the default tap times, (4) allowing it to be further increased by the user to make it work on especially slow connections.