A follow-up to #1474

[achimnol]

What Operating System(s) are you seeing this problem on?

macOS

Which Wayland compositor or X11 Window manager(s) are you using?

No response

WezTerm version

20220903-194523-3bb1ed61

Did you try the latest nightly build to see if the issue is better (or worse!) than your current version?

Yes, and I updated the version box above to show the version of the nightly that I tried

Describe the bug

1. NFD rendering issue with bold texts

After #1474, it renders NFD Hangul characters correctly, but I found that it does only when the font is not bold.

But when it becomes bold:

The last folder is created with Finder, which uses NFD. Other folders are created with mkdir in the terminal, which uses NFC.

2. NFD rendering issue with text selection drags using tmux set mouse on

Also, dragging the text (with tmux's set mouse on) in the terminal makes it garbled with NFD characters while it's fine with NFC characters. It seems fine with the vanilla text selection (with tmux's set mouse off or dragged with the Shift key). This issue happens even with non-bold text as well.

I think the text width calculation logic still misses something with NFD hangul characters.

To Reproduce

test-color.txt test.txt cat the above files in the terminal to reproduce it.

Configuration

local wezterm = require 'wezterm';

return {
    use_ime = true,
    color_scheme = "Rapture",
    window_decorations = "RESIZE",
    native_macos_fullscreen_mode = false,
    font = wezterm.font_with_fallback({
        "JetBrains Mono",
        -- "Menlo",
        "Pretendard",
        -- "Apple SD 산돌고딕 Neo",
        -- "AppleGothic",
    }),
    custom_block_glyphs = false,
    colors = { compose_cursor = "orange" },
    keys = {
      {key="LeftArrow", mods="SUPER", action=wezterm.action{ActivateTabRelative=-1}},
      {key="RightArrow", mods="SUPER", action=wezterm.action{ActivateTabRelative=1}},
      {key="Tab", mods="CTRL|SHIFT", action=wezterm.action{ActivateTabRelative=-1}},
      {key="Tab", mods="CTRL", action=wezterm.action{ActivateTabRelative=1}},
      {key="UpArrow", mods="SUPER", action=wezterm.action{SendString="\x1b[5~"}},   -- PageUp
      {key="DownArrow", mods="SUPER", action=wezterm.action{SendString="\x1b[6~"}}, -- PageDown
      {key="f", mods="SHIFT|SUPER", action="ToggleFullScreen"},
    },
    -- debug_key_events = true,
}

Expected Behavior

No response

Logs

No response

Anything else?

The font used to render the Hangul characters: https://cactus.tistory.com/306

[wez]

I've pushed a commit that seems to help with this; please let me know how it works out!

[achimnol]

The nightly version 20220905-171004-86c9d781 fixes the bold rendering issue at rest. The tmux text selection is vastly improved but still there seems to be minor glitches.

https://user-images.githubusercontent.com/555156/188570708-6f30afcc-9fb4-4749-8d92-f5bf557504aa.mov

[achimnol]

Hm... three more cases:

2. Dragging in the reverse direction with tmux text selection

https://user-images.githubusercontent.com/555156/188572845-2016ce85-50d7-4ef8-87a9-bd55b0e46e7a.mov

3. The cursor position miscalculation with NFD hangul chars

https://user-images.githubusercontent.com/555156/188572859-2d47659d-db63-4949-8604-366f2e174d10.mov

4. Misrendered at first time and correct by itself after text selection trial

Reproduction sample: test-color2.txt

https://user-images.githubusercontent.com/555156/188573437-60c5b352-3aab-4826-869d-d723a572ce81.mov

[achimnol]

• Case 1 and 2 (tmux text selection issue): I'm not sure whether it is due to tmux or wezterm. If it's the tmux issue, I will report the bug there. Please let me know.
• Case 3 (the cursor position issue): happens even without tmux.
• Case 4 (misrendered and self-corrected): happens only inside tmux.

[wez]

For the tmux cases, do those behave the same way if you run tmux inside Terminal.app?

Part of the reason they behave as they do in wezterm is that wezterm shapes runs of text that have the same attributes. The shaper is where the text is effectively composed for this input text.

Since tmux emulates selection by changing the attributes of the cells, that causes the runs of codepoints to be clustered differently; they're no longer an "atomic" run of the codepoints that comprise the filename, but rather separate fragments of the original text.

I'm thinking that we could introduce an option, that defaults to false to avoid a performance penalty for the majority of users, to have wezterm normalize to NFC when a run of text is flushed into the model, around here:

https://github.com/wez/wezterm/blob/e585178d2fb53f5a0773b96da5cfaff016c2d401/term/src/terminalstate/performer.rs#L118-L120

that wouldn't help for the tmux selection scenario where it fragments the text into separate runs, but seems like it would avoid most headaches with NFD in "pure wezterm".

[wez]

I'm trying to think of simple and robust ways to deal with NFC normalization across attribute boundaries, but without success. There are thorny issues when it comes to reasoning about how the codepoints compose or re-order that are hard to reconcile back to the appropriate attributes.

[wez]

Once the nightly build completes, you can set normalize_to_nfc = true in your config to normalize to NFC.

[wez]

I'm not sure that I can reproduce the cursor issue either with or without this config in the currently nightly builds. Could you capture a screen recording of that both inside and outside tmux using wezterm record?

• Launch wezterm. If possible, please use the default terminal size of 80x24 cells as it helps to keep things smaller and easier to manage.
• Inside that terminal run wezterm record to start a recording session.
• Run through your reproduction steps
• Then type exit
• You should see a message like:

*** Finished recording to /var/tmp/wezterm-recording-sF6B3u.cast.txt

• Attach the file that it produced to this issue.

The file is an asciicast (compatible with https://asciinema.org/) and can also be replayed using wezterm replay.

The terminal recording allows me to replicate what is being sent to the terminal without requiring me to install the same applications as you and replicate your configuration for everything.

[achimnol]

Once the nightly build completes, you can set normalize_to_nfc = true in your config to normalize to NFC.

I think this would be a good realistic compromise. There is no big need to "preserve" the original NFD text when copy-and-pasting the text. (Actually, most Korean users want that Apple should change the default behavior from NFD to NFC because this makes many headaches when sharing files with Windows users....)

[achimnol]

Here is the recording: wezterm-recording-DL6TzF.cast.txt

[wez]

Thanks for the recording; replaying it inside Terminal.app also exhibits similarly broken behavior, so I'm inclined to believe that tmux has some issues in this area. I can't solve those in wezterm.

I'd like to focus on case 3 that you mentioned in a comment above, where you said that it happened without tmux. Could you capture a recording of that happening outside of tmux?

[achimnol]

Here is the recording for case 3: wezterm-recording-Djtv4P.cast.txt

[wez]

Is there more that needs to be done here? Could you try the latest release?

[github-actions[bot]]

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.