Feedback wanted: switch default fallback to new regex-based indent plugin

[yioneko]

TL;DR

Help to test the new regex-based indent plugin vim-tmindent as fallback to see if it reduces unexpected indent result. I'm considering to switch the default fallback method to it if the overall feedbacks are positive. Any feedbacks or suggestions are welcome :)

Background

As mentioned in #1, tree-sitter cannot properly handle cases where the syntax tree contains error, especially during editing. Even though I introduced some dedicated handlers for complex situations like them, but I simply cannot cover all the cases, mostly depending on the editing flow of different users. One of the main goal of the last rewrite is fallback method support to help avoid unexpected result caused by tree-sitter error. By default vim builtin indent methods are used as fallback:

• auto: :h 'autoindent', use indent of previous line
• asis: directly returns indent of current line
• cindent: :h cindent(), compute indent as C indentation style

auto and asis are just "stupid" to not consider any language-specific rules, and cindent() only applies for C-style programming language and makes little sense in lua, python, html, or many other languages.

Considering these pitfalls, none of them should be the default option for fallback indent method.

Language specific indentexpr?

That might be like:

require("nvim-treesitter.configs").setup {
  yati = {
    overrides = {
      python = { fallback = function() return vim.fn["GetPythonIndent"]() end }
    }
  }
}

Well, this is reasonable to let the difficult part handled other side. However, there are many problems here:

• Nearly all the current indent ftplugins requires traditional :syntax on to work correctly. For example: https://github.com/Vimjas/vim-python-pep8-indent/blob/60ba5e11a61618c0344e2db190210145083c91f8/indent/python.vim#L69 Running two syntax highlight engines (the one of tree-sitter and the another of vim) simultaneously will slow down the editor and cause highlight conflicts.
• Even if we could found plugins without requirement of :syntax on, should we require users to install them or let the user manually assign the individual indent method? Both will make this plugin "not just work" and lose of control (because of the unpredictable quality of other indent plugins)

Solution

Switch to a generic indent plugin like vim-gindent, I think this is the best fit because:

• We only use it to calculate indent in fallback situation where tree-sitter doesn't work. That is, the plugin doesn't need to be extremely accurate to handle all the edge cases, it is enough to be predictable in common cases like enter on new line, incomplete pairs, etc.
• Not rely on :syntax on at all.
• User only need to install one extra plugin for better support for many languages.

Why create another one?

The idea of vim-gindent is great, but its quality is not that ideal and relies on tree-sitter for "syntax match". The improvements of vim-tmindent mainly includes:

• Compatible with TextMate indentation rule, which is also used in editors like atom, vscode for auto indent. The good parts here are apparent: we can easily copy existing indentation rule from community of other editors (mainly vscode) and the rules made here could also be shared with them.
• More dedicated languages support, compared with only lua, vim and php.
• Optional requirement for tree-sitter, and just leave the hard part to it like comment trimming and language detection.

And it is maintained by myself, it might be easier to fix issues reported from here.

Migration

If users generally feel comfortable with the new regex-based fallback method, vim-tmindent might be turned into the required dependency and the default_fallback option will be deprecated. This is unfortunately another breaking change, but suggestions are also welcome.

[AckslD]

I just recently learned about both yeti and tmindent which seem really great so thanks for making these. I think I might just use tmindent since it seems to do what I want and it honestly seems to me that maybe regex based indentation is better than with treesitter. I love treesitter for highlighting and other stuff but as you're typing stuff you usually have errors and getting correct indentation in this case seems tricky.

I mostly write python and have always used vim-python-pep8-indent which works great but it requires additional regex highlighting which currently does not play well with spell.

Do you see any disadvantages of using just tmindent over yati + tmindent?

One thing I noticed directly while testing where I prefer tmindent over yati is the following lua example:

local x = function()|

where | indicates the cursor. If you press <CR>, tmindent gives you

local x = function()
  |

ie it's indented but yati gives

local x = function()
|

.

Another difference I saw is the following python case:

({}, [|])

where tmindent gives

({}, [
    |])

but yati gives

({}, [
])

In this case I'm not completely sure which one I prefer, since if you intend to type entries of the list you can do that directly with tmindent and then you need to dedent when you're done, but with yati you first indent before typing an entry.

[yioneko]

Thanks for your feedback. There should be no big impact of user experience for using standalone tmindent instead of combining the two, as that is how tmindent is designed: porting indent mechanism used in other popular editors like VSCode. That is also why it is written in vimscript to keep compatibility for original vim to act as a drop-in replacement for builtin indent system and bundled indent plugins.

Regex based indent should work for most of the common cases. However, there are still many rare cases which are hard to address in that approach by design, for example:

local foo = --[[ { ]]-- 1
  | -- many editors will wrongly indent here because they see `{` on the last line

While tree-sitter based indent could gracefully handle such case because it is syntax aware. However, as you pointed out,

as you're typing stuff you usually have errors and getting correct indentation in this case seems tricky.

The accuracy of tree-sitter approach is based on completeness and correctness of syntax tree, which is too ideal for editing. For the first case you listed,

local x = function()
|

Due to the missing end pair, tree-sitter cannot recognize the context region of function, because it might be:

local x = function() end
|

That is why I suggested using snippet plugin to keep the syntax tree error free. Personally I use a function snippet for this case:

af => ["function($1)", "\t$2", "end"]

And I nearly never encounter indent issues like this. This depends on editing habit; tree-sitter itself cannot address this kind of issues as far as I know.

The last case seems confusing for me, because I tested tmindent and it gives me the different result compared to yours:

({}, [
|])

And this is actually my expected result, while yours seems a bug to me. Could you confirm :set indentexpr=tmindent#indentexpr()?

BTW, most of the auto pair plugin provide an option to automatically insert new line on enter between brackets, have you checked it or is there any reason not to enable it?

[AckslD]

Thanks a lot for you reply! :)

And this is actually my expected result, while yours seems a bug to me. Could you confirm :set indentexpr=tmindent#indentexpr()?

You were absolutely right, it wasn't actually enabled in that case. I fixed my setup and now works as expected :)

BTW, most of the auto pair plugin provide an option to automatically insert new line on enter between brackets, have you checked it or is there any reason not to enable it?

Yeah, I've tried using auto pair plugins before but I've always found it distracting. Maybe it's something you get used to but at least for now I prefer without.

[9mm]

Interesting... i stumbled across here after hearing raving reviews about treesitter over and over... so i switched to neovim and am like.............. what is this trash. Literally nothing in any language indents properly. It's maddening. How is this so popular where it cannot properly indent while writing code. I don't even see the point at that point.

I'm going to have this a try. Out of curiosity, what do a lot of popular editors like VSCode do? What editor is the "best" as far as just nailing all the complex indenting rules... emacs?

[xzbdmw]

Interesting... i stumbled across here after hearing raving reviews about treesitter over and over... so i switched to neovim and am like.............. what is this trash. Literally nothing in any language indents properly. It's maddening. How is this so popular where it cannot properly indent while writing code. I don't even see the point at that point.

I'm going to have this a try. Out of curiosity, what do a lot of popular editors like VSCode do? What editor is the "best" as far as just nailing all the complex indenting rules... emacs?

JetBrains never have wrong indent