wezterm icon indicating copy to clipboard operation
wezterm copied to clipboard
verified publisher icon wez

Add support for RTL languages

Open CIAvash opened this issue 4 years ago • 71 comments

Is your feature request related to a problem? Please describe. wezterm cannot display right to left languages correctly. RTL text is not not RTL, and characters that need to be combined, are not.

Describe the solution you'd like Support RTL text. Probably needs bidirectional text handling and text shaping.

Related projects: harfbuzz, FriBidi

Describe alternatives you've considered Konsole and gnome-terminal support RTL languages.

Additional context Image, although the image is comparing Konsole with Alacritty, wezterm works just like Alacritty.

CIAvash avatar May 11 '21 18:05 CIAvash

https://terminal-wg.pages.freedesktop.org/bidi/ has some excellent notes on how to model bidi in terminal emulators.

To make progress, I need to better understand:

  • what is the bare minimum set of features/level of conformance required to make bidi useful in a terminal emulator?
  • how I can reasonably test support when I don't understand the RTL scripts
  • what the feature gaps are between the current set of Rust bidi crates and eg: fribidi (which doesn't currently have Rust bindings, and whose viral LGPL license is probably fine, but potentially fraught with legal concerns in a statically linked application)

wez avatar May 12 '21 02:05 wez

Can pango be an option? Although it has some gtk dependencies, it uses harfbuzz and fribidi. That's probably how gnome-terminal supports RTL.

For testing, let me know if I can help by providing text content.

Also, if you are using harfbuzz, shouldn't Arabic script characters get combined? Currently they don't.

CIAvash avatar May 12 '21 08:05 CIAvash

I forgot to say that (I think) @behdad (creator of harfbuzz) is responsive, if you have questions.

CIAvash avatar May 12 '21 08:05 CIAvash

There is also servo's unicode-bidi. Mentioned in alacritty/alacritty#663.

CIAvash avatar May 18 '21 19:05 CIAvash

There's discussion on https://github.com/kas-gui/kas-text/issues/20 about bidi implementations for Rust.

My impression right now is that the state of bidi in Rust is young and that the easiest path will result in a relatively slow bidi implementation, which isn't ideal: shaping already costs perf in wezterm today. Putting in more work on the promising alternative mentioned in that thread will likely be a better end-state, but will take more effort and that shouldn't be owned by wezterm.

The main constraint I have right now is time: if someone has time and wants to drive this forward, I'm very receptive to seeing wezterm support bidi and helping that person figure out how to integrate it into wezterm.

wez avatar May 19 '21 00:05 wez

I've pushed a commit with what is probably the bare minimum level of support: I'm sure it's wrong in a number of cases, but with this as my test case (borrowed from https://github.com/microsoft/terminal/issues/538#issuecomment-677017322)

Starting wezterm like this to start with the default config, then make the font bigger and turn on bidi mode:

wezterm -n --config font_size=36 --config initial_rows=5 --config initial_cols=30 \
    --config experimental_bidi=true

that's equivalent to running with this config:

return {
   font_size = 36,
   initial_rows = 5,
   initial_cols = 30,
   experimental_bidi = true, -- this is the bit you want to use to try this out
}

Pasting: This is RTL -> عربي فارسی into the terminal:

image

TODO:

  • [ ] Get feedback from people that actually work with RTL languages in the terminal to get a sense of what is actually good/desirable and/or counter-examples to see what to avoid
  • [x] Config option should probably not be a boolean, but instead allow user to specify Disabled, LTR, RTL or Auto-LTR bidi resolution modes
  • [ ] Respect the bidi related escape sequences as discussed in https://terminal-wg.pages.freedesktop.org/bidi/recommendation/escape-sequences.html
  • [ ] DECRLM and other modes; take a look at Hterm as was suggested by j4james below
  • [ ] Consider carefully the default mode for bidi. https://terminal-wg.pages.freedesktop.org/bidi/recommendation/basic-modes.html suggests a default mode that isn't compatible with terminal hardware as pointed out by @j4james in the linked issue. Need to find a good balance between compatibility and things working as users expect.
  • [ ] Currently, wezterm resolves bidi on attributed text runs within a single physical (post-wrapped) line. https://terminal-wg.pages.freedesktop.org/bidi/recommendation/paragraphs.html recommends that it be carried out on the logical line prior to wrapping. That also needs some thought. Right now the context in which bidi is computed in wezterm doesn't have access to that paragraph information. This will be easier to reconcile if someone can provide a good example of how things should look.
  • [x] Turning on bidi mode effectively regresses #478 so we'll need to revisit how to resolve that
  • [x] In the example above, فا renders with a gap between the ligature, whereas what renders in the browser doesn't have a gap. I believe that is because the first portion of that sequence has half-width and since we're monospace the other half is a gap. I don't know what the best way to handle this is.

wez avatar Jan 25 '22 16:01 wez

Note that the terminal-wg bidi document, while giving the impression of being well researched, makes no mention of the DEC RTL sequences from the VT5xx terminals (e.g. DECRLM) and related modes supported by Hebrew terminal emulators like Hterm. IMO those existing modes were much more useful for anyone doing serious RTL development than any of the modern proposals.

j4james avatar Jan 25 '22 17:01 j4james

Thanks James; I'll queue up some more reading/research!

wez avatar Jan 25 '22 18:01 wez

@behdad I don't mean to pounce, but I wonder if you have suggestions specifically on handling the narrower glyphs in فا in a monospace/terminal context; the x_advance in this case is approx. half the monospace cell width. wezterm uses harfbuzz under the covers, but has some logic to override x_advance to make cells line up. Is this particular case best solved simply by using a different font that has wider versions of these glyphs? Or are there some recommended flags/modes for harfbuzz that I should consider?

This is how that same sequence renders in Terminal.app: image

Even if I use the same font (which I think is the SF Arabic font), I still have gaps in my presentation. It feels like something in Terminal.app knows to stretch those ligatures and I wonder if harfbuzz has some way to express that? Or is this just deep magic in Apple's shaper/typography implementation?

(Maybe sort of related: #1333 is a feature request for Devanagari support, which also has some challenging glyph widths for a terminal. Would love to hear your thoughts on that as well!)

I'd also love to hear if you have other recommendations on bidi/rtl support in the context of a terminal?

wez avatar Jan 26 '22 04:01 wez

Currently I can report 2 issues:

  • If you put a number or LTR letter after an RTL letter(with or without space), it becomes LTR. On VTE based terminal, numbers work fine, but if you put an LTR letter, it becomes LTR.
  • Moving cursor position doesn't follow the RTL letter positions, So you can't tell where your'e typing(or changing) a letter.

The other one you mentioned yourself, a space between glyphs that are combined together; I see this problem in VTE based terminal as well, if a non-monospace font is used. If I use a monospace(DejaVu Sans Mono) font it shows correctly(in wezterm and VTE based terminal).

CIAvash avatar Jan 26 '22 06:01 CIAvash

Is this particular case best solved simply by using a different font that has wider versions of these glyphs?

Yes.

It feels like something in Terminal.app knows to stretch those ligatures and I wonder if harfbuzz has some way to express that? Or is this just deep magic in Apple's shaper/typography implementation?

HarfBuzz doesn't know that. I haven't checked Terminal.app. It might be a geometric stretch. You sure it's using the same font?

behdad avatar Jan 26 '22 13:01 behdad

This is how that same sequence renders in Terminal.app: image

Looks obviously a different font.

behdad avatar Jan 26 '22 15:01 behdad

I didn't find exactly the font that Terminal.app is using, but I found that updating my local copy of Cascadia Code and using that looked better: I'll stop chasing that particular dragon :)

wez avatar Jan 26 '22 16:01 wez

Currently I can report 2 issues:

  • If you put a number or LTR letter after an RTL letter(with or without space), it becomes LTR. On VTE based terminal, numbers work fine, but if you put an LTR letter, it becomes LTR.

Could you run: wezterm ls-fonts --text "EXAMPLE" where example is the text sequence you're trying, so that I can see exactly what sequence you mean and also what wezterm thinks it is doing?

  • Moving cursor position doesn't follow the RTL letter positions, So you can't tell where your'e typing(or changing) a letter.

I haven't done anything about cursor positioning or input so far. I don't know how to type this script into the terminal; could you run through how you do that? I'm assuming that you have a particular keyboard/IME configured. Could you walk me through typing a short bit of text (a couple of letters/glyphs) that mixes LTR and RTL so that I can try this for myself and not produce nonsense?

The other one you mentioned yourself, a space between glyphs that are combined together; I see this problem in VTE based terminal as well, if a non-monospace font is used. If I use a monospace(DejaVu Sans Mono) font it shows correctly(in wezterm and VTE based terminal).

I think part of the docs to write up around this will be to suggest a good monospace font. Cascadia Code is another option that at least is monospace, but for which I am not equipped to comment on legibility/usability vs. other Arabic fonts!

wez avatar Jan 26 '22 16:01 wez

Could you run: wezterm ls-fonts --text "EXAMPLE" where example is the text sequence you're trying, so that I can see exactly what sequence you mean and also what wezterm thinks it is doing?

Only letters(متن=text, فارسی=Persian=Farsi), which works fine:

متن فارسی

wezterm ls-fonts --text "متن فارسی" 
RightToLeft
15 ی    \u{6cc}      x_adv=10 glyph=3113 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
13 س    \u{633}      x_adv=10 glyph=3182 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
11 ر    \u{631}      x_adv=10 glyph=1127 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 9 ا    \u{627}      x_adv=10 glyph=3145 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 7 ف    \u{641}      x_adv=10 glyph=3214 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 6      \u{20}       x_adv=10 glyph=1    wezterm.font("Roboto Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/RobotoMono-Regular.ttf, FontConfig
 4 ن    \u{646}      x_adv=10 glyph=3233 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 2 ت    \u{62a}      x_adv=10 glyph=3155 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 0 م    \u{645}      x_adv=10 glyph=3230 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig

Same text with spaces and a number(۲=2) between the words:

متن ۲ فارسی

wezterm ls-fonts --text "متن ۲ فارسی" 
RightToLeft
 6      \u{20}       x_adv=10 glyph=1    wezterm.font("Roboto Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/RobotoMono-Regular.ttf, FontConfig
 4 ن    \u{646}      x_adv=10 glyph=3233 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 2 ت    \u{62a}      x_adv=10 glyph=3155 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 0 م    \u{645}      x_adv=10 glyph=3230 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
LeftToRight
 0 ۲    \u{6f2}      x_adv=10 glyph=1194 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
RightToLeft
 9 ی    \u{6cc}      x_adv=10 glyph=3113 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 7 س    \u{633}      x_adv=10 glyph=3182 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 5 ر    \u{631}      x_adv=10 glyph=1127 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 3 ا    \u{627}      x_adv=10 glyph=3145 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 1 ف    \u{641}      x_adv=10 glyph=3214 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 0      \u{20}       x_adv=10 glyph=1    wezterm.font("Roboto Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/RobotoMono-Regular.ttf, FontConfig

Same text with spaces and a number(2) between the words:

متن 2 فارسی

wezterm ls-fonts --text "متن 2 فارسی" 
RightToLeft
 6      \u{20}       x_adv=10 glyph=1    wezterm.font("Roboto Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/RobotoMono-Regular.ttf, FontConfig
 4 ن    \u{646}      x_adv=10 glyph=3233 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 2 ت    \u{62a}      x_adv=10 glyph=3155 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 0 م    \u{645}      x_adv=10 glyph=3230 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
LeftToRight
 0 2    \u{32}       x_adv=10 glyph=56   wezterm.font("Roboto Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/RobotoMono-Regular.ttf, FontConfig
RightToLeft
 9 ی    \u{6cc}      x_adv=10 glyph=3113 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 7 س    \u{633}      x_adv=10 glyph=3182 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 5 ر    \u{631}      x_adv=10 glyph=1127 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 3 ا    \u{627}      x_adv=10 glyph=3145 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 1 ف    \u{641}      x_adv=10 glyph=3214 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 0      \u{20}       x_adv=10 glyph=1    wezterm.font("Roboto Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/RobotoMono-Regular.ttf, FontConfig

With double quotes the spaces are also misplaced.

Without double quotes: wezterm ls-fonts --text (echo -n متن 2 فارسی)

متن 2 فارسی

I haven't done anything about cursor positioning or input so far. I don't know how to type this script into the terminal; could you run through how you do that? I'm assuming that you have a particular keyboard/IME configured. Could you walk me through typing a short bit of text (a couple of letters/glyphs) that mixes LTR and RTL so that I can try this for myself and not produce nonsense?

I set keyboard layouts in Sway window manager like this:

input * {
    xkb_layout "us,ir"
    xkb_options "grp:shifts_toggle,compose:caps"
}

And toggle between English and Persian.

In X, I think it's with this command: setxkbmap -layout us,ir -option grp:shifts_toggle or xorg config:

    Option "XkbLayout" "us,ir"
    Option "XkbOptions" "grp:shifts_toggle"

You can use online virtual keyboards: https://www.branah.com/farsi - With this you can switch between Persian and English https://www.lexilogos.com/keyboard/persian.htm - This one has the pronunciation of letters

So for typing "متن ۱ فارسی" in Persian keyboard layout: You would hit these keys: l j k SPACE 1 SPACE t h v s d For "متن RTL و متن LTR": l j k SPACE R T L SPACE , SPACE ‍l j k SPACE L T R Last text on its own(Beginning with RTL letters):

متن RTL و متن LTR

Some random text samples: From Persian alphabet:

الفبای فارسی یا الفبای فارسی-عربی شاملِ ۳۲ حرف است که از الفبای عربی اقتباس‌شده است.

From English language:

اِنگلیسی (به انگلیسی: English، ‎/ˈɪŋɡlɪʃ/‎) یک زبان طبیعی از خانواده زبانی زبان‌های هندواروپایی از شاخه زبان‌های ژرمنی غربی است که اولین بار در انگلستان در عهد آنگلوساکسون‌ها مورد تکلم قرار گرفت و انگلیسی باستان شکل گرفت.

From Persian language

There are several letters generally only used in Arabic loanwords. These letters are pronounced the same as similar Persian letters. For example, there are four functionally identical letters for /z/ (ز ذ ض ظ), three letters for /s/ (س ص ث), two letters for /t/ (ط ت), two letters for /h/ (ح ه). On the other hand, there are four letters that don't exist in Arabic پ چ ژ گ.

I think part of the docs to write up around this will be to suggest a good monospace font. Cascadia Code is another option that at least is monospace, but for which I am not equipped to comment on legibility/usability vs. other Arabic fonts!

I took a look at Cascadia Code, it seems it's the font Microsoft uses for Windows terminal. In my opinion it doesn't look good, letters get stretched and are sometimes hard to read. There may be better fonts, but I haven't searched for one.

CIAvash avatar Jan 26 '22 20:01 CIAvash

There is Vazir Code fonts, the Vazir Code Hack seems to look better.

CIAvash avatar Jan 27 '22 04:01 CIAvash

Thanks for this: it gives me something to play with and reason about!

wez avatar Jan 27 '22 12:01 wez

Thank you for working on this.

CIAvash avatar Jan 27 '22 13:01 CIAvash

@behdad At the moment, I use the UBA to produce runs of the various embedding levels (to determine the direction) and feed each of those to harfbuzz without any bidi reordering. https://harfbuzz.github.io/what-harfbuzz-doesnt-do.html doesn't explicitly say which parts of the bidi algorithm should be applied pre/post shaping. Do you have recommendations about this?

I'm trying to figure out what I'm doing wrong for this example; the first grouping results in the space being reordered to the left and the last grouping has it reordered to the right. When wezterm renders these, it will render them starting from x=0 in the order they are listed below, incrementing x by the x_advance. The result is that there is no space between these runs, only around the edges.

; wezterm ls-fonts --text "متن ۲ فارسی"
RightToLeft
 6      \u{20}       x_adv=10 glyph=1    wezterm.font("Roboto Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/RobotoMono-Regular.ttf, FontConfig
 4 ن    \u{646}      x_adv=10 glyph=3233 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 2 ت    \u{62a}      x_adv=10 glyph=3155 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 0 م    \u{645}      x_adv=10 glyph=3230 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
LeftToRight
 0 ۲    \u{6f2}      x_adv=10 glyph=1194 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
RightToLeft
 9 ی    \u{6cc}      x_adv=10 glyph=3113 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 7 س    \u{633}      x_adv=10 glyph=3182 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 5 ر    \u{631}      x_adv=10 glyph=1127 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 3 ا    \u{627}      x_adv=10 glyph=3145 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 1 ف    \u{641}      x_adv=10 glyph=3214 wezterm.font("DejaVu Sans Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/DejaVuSansMono.ttf, FontConfig
 0      \u{20}       x_adv=10 glyph=1    wezterm.font("Roboto Mono", {weight="Regular", stretch="Normal", italic=false})
                                      /usr/share/fonts/TTF/RobotoMono-Regular.ttf, FontConfig

wez avatar Jan 27 '22 15:01 wez

I'm far removed from bidi algorithm right now to know what the expected output is.

behdad avatar Jan 27 '22 20:01 behdad

I didn't find exactly the font that Terminal.app is using,

The Arabic text in that screenshot is set in Courier New.

khaledhosny avatar Jan 28 '22 23:01 khaledhosny

without any bidi reordering

You need to reorder the runs, but without reversion the characters in RTL runs.

khaledhosny avatar Jan 28 '22 23:01 khaledhosny

Thanks @khaledhosny! OK, the current state of main appears to be at a similar level of support as Terminal.app to my untrained eye for the case of typing متن ۱ فارسی using the directions provided above.

  • The cursor movement is not BIDI aware
  • The text is reordered as it is typed, and renders in the same order as it appears in Chrome, but left aligned in my shell
  • Copying and pasting a subset of the line doesn't account for visual/logical remapping

Beyond the selection/copy/paste stuff, I think the next level of support would be to consider actual bidi-aware programs and see where wezterm has gaps; eg: supporting DECRLM and BIDI related escape sequences. I don't know what software has support for those things, or how it might be configured so I would love to hear about that.

I'm also unsure about eg: cating a text file with RTL content; would you expect that to right-justify RTL lines by default?

wez avatar Jan 29 '22 13:01 wez

I'm also unsure about eg: cating a text file with RTL content; would you expect that to right-justify RTL lines by default?

In VTE-based terminals the text is left-jusified. If you pipe text to fribidi command line tool, it right-justifies the text, but with a fixed width.

I personally think it should be right-justified. Because that feels right, and that's how it is in GUI softwares. The width of the text is the same as the width of widget/window. ‌‌But each program seems to do it differently(per line, per paragraph or whole text).

A closer software to terminal is Emacs, it aligns the text to the right, I think only in text modes. And it does so differently, it doesn't do it per line, but per paragraph. So if paragraph starts with LTR letters, the whole paragraph is LTR and vice versa.

And I just tried to see how Emacs behaves in terminal, the text alignment behavior is the same, but it seems it tries to make it RTL again, resulting in a change of direction! So the RTL text becomes LTR(the letters themselves go from left to right!).

CIAvash avatar Jan 29 '22 19:01 CIAvash

And I just tried to see how Emacs behaves in terminal, the text alignment behavior is the same, but it seems it tries to make it RTL again, resulting in a change of direction! So the RTL text becomes LTR(the letters themselves go from left to right!).

This is because it's designed to work in a standard terminal which doesn't reorder the display of RTL characters. Once the terminal decides to take responsibility for the character ordering, it makes it impossible for RTL/bidi-aware software to work.

You need to pick a side: you can't support actual RTL software and non-RTL software at the same time. Best you can probably do is provide an option that lets the user choose.

Some terminals that do RTL reordering will also have an escape sequence to disable that functionality, which apps like Emacs could potentially use. But I don't think there's a standard for that.

j4james avatar Jan 29 '22 19:01 j4james

Emacs has some variables and functions which let you customize and change things. If I set bidi-display-reordering to nil then the text is displayed correctly, but it will be left-justified. I can also call set-justification-right to make it right-justified, but it seems to follow a limited width.

Emacs also has a variable called bidi-directional-controls-chars with value "\x202a-\x202e\x2066-\x2069".

CIAvash avatar Jan 29 '22 20:01 CIAvash

I just played a little bit with mlterm and it magically swaps my shell prompt to RTL when typing in the farsi text from above. It's pretty cool but definitely seems like it would be fraught with problems for compatibility.

https://terminal-wg.pages.freedesktop.org/bidi/recommendation/escape-sequences.html mentions a couple of escape sequences that are present in ECMA 48; BDSM (!), SCP and SPD that influence this behavior.

The main thing that has noticeable effect appears to be the SCP sequence: CSI 2 SPACE k to set to RTL or CSI 1 SPACE k to set to LTR. Emitting CSI 2 SPACE k causes the shell and all subsequent output to mirror similar to how mlterm looks when it sees RTL text in the line, but does it regardless of whether there is RTL text.

As James noted, it doesn't mention DECRLM. In that RTL mirrored mode, it doesn't appear necessary to manipulate the cursor movement, as it is effectively automatically flipped.

This is a screenshot of VTE: image

wez avatar Jan 29 '22 20:01 wez

mlterm doesn't support those SCP sequences (CSI 2 SPACE k)

wez avatar Jan 29 '22 20:01 wez

Some of the other RTL escape sequences I'm aware of include:

  • CSI ?37h - mirror mode (I think this essentially flips the entire screen)
  • CSI ?39h - implicit RTL mode (I think this is for enabling the bidi rendering algorithm)
  • CSI ?77096h - this is Mintty's mode for enabling bidi rendering
  • OSC 50;BidiRenderingEnabled=true ST - this is Konsole's method for enabling bidi rendering

I don't know the details of how they worked though.

j4james avatar Jan 29 '22 20:01 j4james

I found this gist with a summary of bidi support in various apps: https://gist.github.com/XVilka/a0e49e1c65370ba11c17

wez avatar Jan 30 '22 00:01 wez