Arabic doesn't render correctly

[typoman]

Hi, I use vim inside the terminal and it renders Arabic correctly. But I'm missing MacVim features. I was wondering if MacVim is using CoreText, then why Arabic is rendered incorrectly?

Take the following words for example, if you check the vim 8 in terminal it should display it as you can see it in your browser (if you have already executed set noarabic).

عربی فارسی

‌But in MacVim, the Arabic text is rendered from left to right. I saw there is an issue already posted here #696, but was it fixed?

[ychin]

Sorry but I don't speak Arabic so it's a little hard for me to tell what's correct and what's not. Can you take screenshots of what you expected to happen (vim 8 terminal), vs what you saw in MacVim?

Also, when you say Vim 8 in terminal, what Vim do you mean specifically? Do you mean system-provided Vim? Or MacVim but in terminal?

[typoman]

I hope I can show how to make it work. I don't use MacVim in the terminal, I use the MacVim.app (Custom Version 8.1.280 (151)). This is the screenshot from the Vim 8 in the terminal. Which is the expected behavior:

Text which I used to make the screenshot: فارسی English

The same text in MacVim:

Also, I needed to change the font in the MacVim to the New Courier because, in the terminal, the fallback font for Arabic is New Courier, but in MacVim it doesn't happen and goes to a font I don't recognzie. As you can see the order of characters for Arabic has become reversed in the MacVim. Arabic is right to left, and in MacVim it's being displayed as left to right.

I have to add that previously if I build the Vim 8 with the +arabic and +farsi feature in the terminal, the Arabic also was displayed incorrectly. But if I build it without the -arabic feature, the arabic in the terminal Vim displayed correctly. So finally I realized I could make the vim in terminal work with +arabic only if the vim recognized the terminal is bidi. So even by having the +arabic feature in the vim inside the terminal, I added the set termbidi in my vimrc and Arabic started to show correctly. I think MacVim also reads the same vimrc file because settings show up the same. I hope this helps to find a solution?

My Vim version (not the MacVim):

~$ vim --version
VIM - Vi IMproved 8.1 (2018 May 18, compiled Dec  2 2018 03:52:24)
macOS version
Included patches: 1-556
Compiled by [email protected]
Huge version without GUI.  Features included (+) or not (-):
+acl               +extra_search      +mouse_netterm     +tag_old_static
+arabic            +farsi             +mouse_sgr         -tag_any_white
+autocmd           +file_in_path      -mouse_sysmouse    -tcl
+autochdir         +find_in_path      +mouse_urxvt       +termguicolors
-autoservername    +float             +mouse_xterm       +terminal
-balloon_eval      +folding           +multi_byte        +terminfo
+balloon_eval_term -footer            +multi_lang        +termresponse
-browse            +fork()            -mzscheme          +textobjects
++builtin_terms    -gettext           +netbeans_intg     +timers
+byte_offset       -hangul_input      +num64             +title
+channel           +iconv             +packages          -toolbar
+cindent           +insert_expand     +path_extra        +user_commands
-clientserver      +job               -perl              +vartabs
+clipboard         +jumplist          +persistent_undo   +vertsplit
+cmdline_compl     +keymap            +postscript        +virtualedit
+cmdline_hist      +lambda            +printer           +visual
+cmdline_info      +langmap           +profile           +visualextra
+comments          +libcall           -python            +viminfo
+conceal           +linebreak         +python3           +vreplace
+cryptv            +lispindent        +quickfix          +wildignore
-cscope            +listcmds          +reltime           +wildmenu
+cursorbind        +localmap          +rightleft         +windows
+cursorshape       -lua               -ruby              +writebackup
+dialog_con        +menu              +scrollbind        -X11
+diff              +mksession         +signs             -xfontset
+digraphs          +modify_fname      +smartindent       -xim
-dnd               +mouse             +startuptime       -xpm
-ebcdic            -mouseshape        +statusline        -xsmp
+emacs_tags        +mouse_dec         -sun_workshop      -xterm_clipboard
+eval              -mouse_gpm         +syntax            -xterm_save
+ex_extra          -mouse_jsbterm     +tag_binary        
   system vimrc file: "$VIM/vimrc"
     user vimrc file: "$HOME/.vimrc"
 2nd user vimrc file: "~/.vim/vimrc"
      user exrc file: "$HOME/.exrc"
       defaults file: "$VIMRUNTIME/defaults.vim"
  fall-back for $VIM: "/usr/local/share/vim"
Compilation: gcc -c -I. -Iproto -DHAVE_CONFIG_H   -DMACOS_X -DMACOS_X_DARWIN  -g -O2 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=1       
Linking: gcc   -L/usr/local/lib -o vim        -lm -lncurses  -liconv -framework AppKit     -L/usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/lib/python3.7/config-3.7m-darwin -lpython3.7m -ldl -framework CoreFoundation      

[typoman]

I'm very interested to make this work. Could anyone please direct me where should I look in the source and what is the main part that takes care of glyph generation and layout?

[ychin]

All MacVim's code are in here: https://github.com/macvim-dev/macvim/tree/master/src/MacVim

There are two text views: MMCoreTextView (used when Core Text renderer is on, which is default), and MMTextView which uses the OS native text view (used when Core Text renderer is off). There's also MMTypeSetter which may be part of what triggers this but I'm not sure. I did try to reproduce this and noticed that this bug happens in both Core Text renderer and non-Core Text one.

[ychin]

Hi @typoman I want to circle back to this since there are some discussions on this topic in upstream Vim. Is this still an issue for you and did you get around it? Are you just calling set arabic for now? Or just use Apple Terminal to handle it?