Cannot input many unicode characters on windows on some applications

[chtenb]

What Operating System(s) are you seeing this problem on?

Windows

Which Wayland compositor or X11 Window manager(s) are you using?

No response

WezTerm version

20240325-093507-e5ac32f2

Did you try the latest nightly build to see if the issue is better (or worse!) than your current version?

Yes, and I updated the version box above to show the version of the nightly that I tried

Describe the bug

I cannot input certain characters on wezterm in helix or nushell. If I try, nothing happens. Nothing is being inserted. Unicode characters that cannot be input: 🎉𝓒 Unicode characters that are not a problem: ✅ On other terminal emulators, like cmd.exe, this is issue does not occur.

To Reproduce

Copy 🎉 or 𝓒 to your clipboard and try to insert it on a wezterm instance on windows while running nushell or helix.

Configuration

no config

Expected Behavior

I expected the same behavior as on cmd.exe, where the unicode characters are inserted as expected.

Logs

No response

Anything else?

• Possibly relevant to note that both helix and nushell have a dependency on crossterm.
• I tried setting allow_win32_input_mode to false, but it makes things even worse :)

[chtenb]

I just found that this exact scenario does not work on WSL either, albeit with different symptoms. In WSL on cmd.exe typing unicode in helix just works. In Wezterm however, I only see replacement glyphs trying to type 𝓒:

This used to work in WSL, strangely enough, as can be observed from the upper sentence. I did receive a windows update recently, which could be related.

Here is the binary dump of what is actually inserted in the terminal:

> echo '����' | into binary                                                                                                                    04/04/2024 12:02:27 PM PM
Length: 12 (0xc) bytes | printable whitespace ascii_other non_ascii
00000000:   ef bf bd ef  bf bd ef bf  bd ef bf bd

[chtenb]

As stated, this issue does not occur in other terminal emulators, just on wezterm.

[oOosys]

May you show the same input/output as screenshot in one of the other terminal emulators?

[oOosys]

What you can do to investigate this behavior is to find which font file delivers the glyphs and then put this file into the list of fallback fonts wezterm will use in case the specified default font does not come with required glyph. If I understand it right ezterm allows you to decide about the order of fallback font files used excluding these ones not listed ... it is a feature which makes it necessary to specify the font files to use in the configuration.

[chtenb]

It's not a font issue as you can see in the screenshot I posted. There isn't much to show about the other terminal emulators. It just works.

[oOosys]

Have you tried to put the font which provides the missing glyphs to the list of wezterm fallback fonts? With what effect?

[chtenb]

It's not a font problem, because nothing is inserted. And on WSL something is inserted, but not a valid single character. If it were a font problem, you would see a single replacement glyph.

So probably some characters are stripped out, possibly by conpty. I'm unsure why the effect is different when running a wsl shell. I'm also unsure why other terminal emulators do not have this problem. Possibly because they interact differently, or with a different version of conpty? I'm not well versed in terminal emulator technicalities, this is mostly handwaving from my part.

[oOosys]

So you haven't tried to put the font to the list of wezterm fallback fonts? Sometimes it is a good idea to try also what seems not to make sense ... It turns often out that one think it does not make sense because the assumptions which lead to this conclusion were wrong ... so just trying something what does not make sense gives sometimes surprising results leading finally to a solution.

[chtenb]

I tried it, it doesn't help.

Two other things I noticed:

• when turning off allow_win32_input_mode, it also inserts ���� on regular windows, just like on WSL.
• On WSL, some key combinations, like C-/, are also not coming through

[chtenb]

I discovered that the virus scanner on my PC silently messed up the nightly wezterm installer without me noticing. I think that somehow caused the nightly installer to fail silently for me. I have now installed wezterm "manually" by downloading and extracting the zip file, and it turns out that the nightly version doesn't have this problem after all! 🎉

I guess it was conpty after all, since that has been updated recently https://github.com/wez/wezterm/commit/e7fe7c07da2da071d75c88ff3ae7a3c0e74d4f1f Apologies for the noise.

[chtenb]

I spoke too soon. It is only fixed for WSL.

[wez]

In the broken scenario, what does the process tree look like, and what does the working process tree look like?

wezterm -> cmd -> helix ? wezterm -> wsl ?

are you using wezterm ssh there at all? git bash vs. msys vs. cygwin vs. wsl or other?

Running through https://wezfurlong.org/wezterm/troubleshooting.html#debugging-keyboard-related-issues might be useful; ISTR that there is a some debug output that that enables that will show you what wezterm is trying to paste in.

[chtenb]

Thank you for taking the time to respond!

I've configured default_prog = { 'nu' }. This is what task manager and procmon show respectively:

Is that helpful, or are you looking for other information?

I'm not using wezterm ssh. Should I?

git bash vs. msys vs. cygwin vs. wsl or other?

What do you mean by this?

I'm trying to get the right debug log messages, but everytime I open the debug overlay, the history of log messages is very short, only showing the keys I used to open the debug log.

Interestingly, inserting 𝓒 into the debug REPL also does not work, albeit I do see something in this case.1

Generating these log message:

09:56:33.739 INFO wezterm_gui::termwindow::keyevent > key_event KeyEvent { key: Char('\u{8}'), modifiers: NONE, leds: (empty), repeat_count: 1, key_is_down: true, raw: Some(RawKeyEvent { key: Physical(Backspace), modifiers: NONE, leds: (empty), phys_code: Some(Backspace), raw_code: 8, scan_code: 14, repeat_count: 1, key_is_down: true, handled: Handled(false) }), win32_uni_char: Some('\u{8}') }
09:56:33.739 INFO wezterm_gui::termwindow::keyevent > send to pane DOWN key=Backspace mods=NONE
09:56:33.844 INFO wezterm_gui::termwindow::keyevent > key_event RawKeyEvent { key: Physical(Backspace), modifiers: NONE, leds: (empty), phys_code: Some(Backspace), raw_code: 8, scan_code: 14, repeat_count: 1, key_is_down: false, handled: Handled(false) }
09:56:33.845 INFO wezterm_gui::termwindow::keyevent > key_event KeyEvent { key: Char('\u{8}'), modifiers: NONE, leds: (empty), repeat_count: 1, key_is_down: false, raw: Some(RawKeyEvent { key: Physical(Backspace), modifiers: NONE, leds: (empty), phys_code: Some(Backspace), raw_code: 8, scan_code: 14, repeat_count: 1, key_is_down: false, handled: Handled(false) }), win32_uni_char: Some('\u{8}') }
09:56:33.846 INFO wezterm_gui::termwindow::keyevent > send to pane UP key=Backspace mods=NONE
09:56:34.691 INFO wezterm_gui::termwindow::keyevent > key_event RawKeyEvent { key: Physical(Return), modifiers: NONE, leds: (empty), phys_code: Some(Return), raw_code: 13, scan_code: 28, repeat_count: 1, key_is_down: true, handled: Handled(false) }
09:56:34.691 INFO wezterm_gui::termwindow::keyevent > key_event KeyEvent { key: Char('\r'), modifiers: NONE, leds: (empty), repeat_count: 1, key_is_down: true, raw: Some(RawKeyEvent { key: Physical(Return), modifiers: NONE, leds: (empty), phys_code: Some(Return), raw_code: 13, scan_code: 28, repeat_count: 1, key_is_down: true, handled: Handled(false) }), win32_uni_char: Some('\r') }
09:56:34.691 INFO wezterm_gui::termwindow::keyevent > send to pane DOWN key=Enter mods=NONE