Enhancement request: Proper RTL movement

[yousufinternet]

#Issue Type

• Enhancement request

Emacs version: GNU Emacs 25.2.1 Operating System: Arch Linux with the latest updates Evil version: 1.2.12 Evil installation type: Manual Graphical/Terminal: Graphical Tested in a clean environment: Yes

Reproduction steps

• Start Emacs
• Type some Arabic (Farsi, Hebrew or Urdu) text so that it appears from the right to the left, for example:

هذا نص عربي للتجربة

• Try moving (using right and left arrow keys, or 'h' and 'l' keys) while in normal mode.

Expected behavior

The movement keys should move the cursor to the left when pressing the left arrow key, and to the right when pressing the right arrow key, while on a RTL text.

Actual behavior

The keys move the cursor to opposite of the direction they are supposed to move the cursor in while in a RTL text.

Further notes

Please note that I have only faced this problem when trying to move by characters, other movements that depend on text objects (i.e. pressing 'e' moves correctly to the end of the word on the left side of the cursor) behave correctly.

I hope that there is a quick workaround that makes evil recognise different language environments and reverse its behaviour accordingly.

Thanks and Best Regards

[wasamasa]

You mention that this happens both when using hjkl and arrow keys. This is a clear indicator that your issue is not Evil-related as Evil doesn't cover the arrow keys. As I can reproduce the same problem in emacs -Q with the arrow keys, I'm closing this. Feel free to M-x report-emacs-bug and convince Eli Zaretskii (the person who implemented bidi support) why the current behavior is wrong.

[yousufinternet]

That is strange because I can't reproduce the issue on emacs -Q. if I run emacs -Q, open an empty file (C-x C-f testfile), and type some RTL text, and move using the arrow keys it moves as it is supposed to move, so I am not sure how you were able to reproduce the issue, but my guess is that you used the lisp evaluation buffer to write the text, which does not support bidi text.

Please note that the problem only occurs in evil normal mode, and not insert mode, that is why I thought that it is evil related, and I am now using emacs without evil, and it's behaving correctly with bidi text.

Thanks for the comment and the suggestion.

[wasamasa]

Indeed, I tried it in *scratch* only because you didn't mention using a specific buffer in the repro. In a text mode buffer I can repro a difference between cursor keys in normal and Emacs state which means that Evil is using its own commands for cursor keys after all.

In vanilla Emacs, C-f and <right> are bound to two different commands, one that isn't bidi-aware and one that is. In Evil, l and <right> are bound to the same bidi-unaware command. The question here is how to deal with this, for that I'd try observing how Vim does things and mirror it if it makes sense, otherwise I'd steal Emacs behavior.

[yousufinternet]

My mistake, I should have clarified that. As far as I know vim/nvim bidi support is very poor, if any at all, since that was one of the reasons that got me to switch to emacs, any way I am no expert, so I will be awaiting your comments :-)

[wasamasa]

From a quick test with :set arabic in Vim, both hjkl and arrow keys move in the logical direction (as in, the direction the key points to) whereas word movement goes in text direction (as in the direction the text flows). This is opposed to vanilla Emacs where C-f moves in text direction and the arrow keys in logical direction. I'm not sure which of the two to prefer here as I haven't ever authored RTL texts or localized websites for RTL scripts. I can see the point in supporting both styles like vanilla Emacs does, but also in reducing confusion by making both follow logical direction.

[yousufinternet]

Cool, I don't know about your plans about how to resolve this, but I was hopping at least that the desired behaviour for hjkl and arrow keys could be set by some option in future updates to be moving the cursor in the logical direction or in the text flow direction as the user desires.

I have tried vim with :set arabic, and it has the same behaviour in normal and insert mode evil has in normal mode (but not insert mode), the arrow keys and hjkl move to the opposite direction of the text flow as you said, while text object movements move normally in the text flow direction.

[yousufinternet]

Adding this to my .emacs gives me the behaviour I want:

(define-key evil-normal-state-map "h" 'left-char)
(define-key evil-normal-state-map "l" 'right-char)
(define-key evil-normal-state-map [left] 'left-char)
(define-key evil-normal-state-map [right] 'right-char)
(define-key evil-visual-state-map "h" 'left-char)
(define-key evil-visual-state-map "l" 'right-char)
(define-key evil-visual-state-map [left] 'left-char)
(define-key evil-visual-state-map [right] 'right-char)

as you said, the commands forward-char and backward-char are not bidi-aware.. your comments helped me find the correct way to think about his issue, thanks.

[wasamasa]

The above workaround could be made a bit shorter by unbinding the keys, so that Emacs falls back to the (non-Evil) defaults:

(define-key evil-motion-state-map "h" nil)
(define-key evil-motion-state-map "l" nil)
(define-key evil-motion-state-map [left] nil)
(define-key evil-motion-state-map [right] nil)

I'd expect it to break something (like repetition or operator usage), but from short testing nothing seems to be affected.

[yousufinternet]

Adding this to my .emacs gives me the behaviour I want:

(define-key evil-normal-state-map "h" 'left-char)
(define-key evil-normal-state-map "l" 'right-char)
(define-key evil-normal-state-map [left] 'left-char)
(define-key evil-normal-state-map [right] 'right-char)
(define-key evil-visual-state-map "h" 'left-char)
(define-key evil-visual-state-map "l" 'right-char)
(define-key evil-visual-state-map [left] 'left-char)
(define-key evil-visual-state-map [right] 'right-char)

as you said, the commands forward-char and backward-char are not bidi-aware.. your comments helped me find the correct way to think about his issue, thanks.

I had to add: (setq visual-order-cursor-movement t)

To get this working again after recent updates

[iyefrat]

Hey, I'm currently writing a bidi module for Doom Emacs, and I came across this issue. I am a Hebrew speaker and want to make an argument for what the default behaviour should be:

I think the default Emacs behaviour is basically correct, and propose that evil adds evil-right-char and evil-left-char motions that mirror the directions of right-char and left-char and are bound to the arrow keys (and follow the vim convention of stopping at line boundaries, unlike the Emacs char movement commands).

I think trying to emulate what Vim does here is a mistake. Vim has the unfortunate task of supporting bidi in mostly terminal environments, which is kind of a nightmare, mainly because you have two applications that have to negotiate the text display, the emulator and the TUI app. Due to this, many terminals deal with this problem differently, and there is no real consistent behaviour. There is a proposed standard, but it's still a draft and not exactly universally embraced (and also stalled it seems. last update was two years ago). The terminal I'm using, Tilix, implemented this, which renders the built in :set rightleft behaviour in vim irrelevant since text gets displayed correctly when it's off and incorrectly when it's on.

I think that vim motions make sense in logical order, since they are all defined with relation to the logical order. But the arrow keys are strongly tied to the visual order, and it's useful to have the option. In vim it seems that the arrow keys are just bound to the same thing as the h/l keys, but again I don't think that trying to copy what vim does here is a good idea, since it wasn't originally written with bidi in mind, and the bidi configuration is trying to grapple with the nightmare of bidi terminal text display.

I'd be happy to send a PR to implement this if you want.

[tomdl89]

@iyefrat personally I think everything you wrote sounds sensible. However, I know next-to-nothing about RTL text, so I'm not sure my opinion is worth much. I'd like to think others may chime in, but this issue is old enough that only us 4 participants are likely to be aware of it. I'll happily review a draft PR, but I'm inclined to say that thorough tests and documentation would be essential for me to merge it. Unless wasamasa has much better bidi familiarity and a more liberal attitude towards this change, that's as good as I can offer, as it's just us two maintaining evil.

[wasamasa]

I haven't learned anything new about bidi text in that time, but I'm more confident about new code not breaking anything existing (there are far less people using bidi and the left/right char commands are mostly a superset of the forward/backward char ones). With regards to tests I agree with @tomdl89, they need to exist to prove that the new commands don't do anything funny (like hitting an infinite loop when trying to move beyond the edge of a bidi line).

[iyefrat]

Alright, great. I'll start looking into it, hopefully I'll have a PR to throw your way soon:tm:.

[JonatanSahar]

Hi, has there been any progress with this since? I'm writing hebrew on Doom and I'm still getting the same behavior (l in evil-mode moves cursor to left and h to the right)

[wasamasa]

No. The intersection between people who understand Evil and people who understand bidi is vanishingly small.

[JonatanSahar]

Adding this to my .emacs gives me the behaviour I want:

(define-key evil-normal-state-map "h" 'left-char)
(define-key evil-normal-state-map "l" 'right-char)
(define-key evil-normal-state-map [left] 'left-char)
(define-key evil-normal-state-map [right] 'right-char)
(define-key evil-visual-state-map "h" 'left-char)
(define-key evil-visual-state-map "l" 'right-char)
(define-key evil-visual-state-map [left] 'left-char)
(define-key evil-visual-state-map [right] 'right-char)

as you said, the commands forward-char and backward-char are not bidi-aware.. your comments helped me find the correct way to think about his issue, thanks.

I had to add: (setq visual-order-cursor-movement t)

To get this working again after recent updates

That actually doesn't work for me (on Doom), have there been any more changes that you needed to do?

[yousufinternet]

Make sure to (setq visual-order-cursor-movement t)