[WIP] Better Neorg Performance

[vhyrro]

Buckle up, this PR is quite the crazy one.

Why?

I almost feel obliged to make this PR as Neorg can sometimes straight up halt on 12K or so line files, which is honestly unacceptable. I went digging into the root cause of the problem and found something very interesting indeed - the problem is vim.treesitter and its highlight functionality.

It turns out that on large files the highlighter slows down to a crawl, mainly because of the following reason: highlights are reparsed every. single. time the buffer is redrawn, all extmarks are ephemeral (they disappear on redraw). I'm unsure if this is a legacy thing, or if it's there for a reason? Either way for more complex queries this makes even the smallest text change sluggish. Neovim is legitimately great at handling many many extmarks at once, so I'm not sure why we only apply a few and keep reapplying them constantly.

Paths we can take

At this point there are two paths that I can go down:

• Modify vim.treesitter and stop it from doing constant rehighlights
• Not use highlights.scm and instead roll a custom implementation for Neorg specifically (perhaps in a core.highlighter module)

I'm inclined to go with the custom implementation for the following reasons:

• I very highly suspect that changing anything in vim.treesitter.highlighter will break at least two existing parsers and anger at least 3 people on the planet :joy: - I really don't want to accidentally break half of the parser ecosystem here
• Neorg already has a very solidly built incremental updating system used in core.norg.concealer and core.syntax - this can easily be extended to apply highlights too!

It's not just Treesitter though

Neorg isn't some holy and perfect thing - I'll be extending and expanding on the incremental system in core.norg.concealer to be even faster, will be removing vim.schedules in many places and making the module system less prone to breakages and some other things on top. Prepare for bLaZinG fAst:rocket: :rocket: :rocket: :exploding_head: speeds.

[vhyrro]

:thinking: messed up the rebase lol

[vhyrro]

There we go :)

[vhyrro]

More news:

I've spent wayyy too much time looking into the indent engine at this point - for those that were unaware the indent engine performs some rather expensive calculations and slows down Neorg while editing, mostly due to needing to reparse the current buffer to run a query on it.

I've tried it all at this point, but I'm afraid without an insanely complex system there's no way I'm doing this properly. I don't know what sort of incremental updating treesitter is doing, but whatever it is, it's pretty hot crap lol. If it were really incremental updating of the syntax tree it would not take 4 seconds to reparse a 450K LOC buffer that had a single line at the bottom changed.

I'm not up to the task to do this now, however here is an idea for the indent system:

• Use treesitter strictly to build up a "context" struct that will serve as a guideline for the indent engine (use treesitter's tree_changed callbacks to make it faster too)
• Use normal regex based indenting with the context struct as an additional source of information.

The drawbacks will definitely be:

• Complex system
• Difficult to customize

I'm a little stumped on what to do, but if I come to some revelation I'll be sure to write it here.

[akinsho]

@vhyrro just wanted to point you in the direction of this PR 👉🏿 https://github.com/neovim/neovim/pull/18109 I don't know how much this relates to the issues you are seeing and I definitely haven't read through your PR nor do I understand the implications of the linked PR exactly but based on what you've said it seems like it could be connected and watching out for the PR or testing it out could have an impact on what you are doing?

EDIT: I know GitHub will do the whole reference link thing so if someone on the other side sees this reference and thinks it's not accurate apologies in advance. Also found: https://github.com/neovim/neovim/issues/18108

[akinsho]

Also, FWIW treesitter is avowedly a very good and fast solution to this exact problem at least based on the numerous talks I've seen from the author, I think if there are issues with its integration in nvim core rather than rolling a custom implementation it's worth working with upstream to resolve any issue if possible regardless of breakages since ultimately it's still considered experimental and this will inevitably end up being a problem for lots of people if performance issues aren't resolved at this unstable stage

Apologies in advance I've fully just butted in on both issues, was just interested and happened to have seen that issue then your pr

[vhyrro]

Hey @akinsho thanks for the heads up! What sort of trickery are you using to find those PRs?? I can never find them myself 😄

Glad to hear that something is being done on that front, hopefully it'll make things speedier. Also perhaps you're right about contributing to upstream instead, I don't know why I'm so afraid of contributing to core lol

[akinsho]

@vhyrro 😆 I just track PRs and issues in the nvim repository to a probably pretty unhealthy degree tbh 😅. I've also just been curious about treesitter for many years, so follow its integration in nvim closely.

Re. contributing to core I definitely get what you mean, although in my case the concern is definitely more about spending even more time on my editor since I definitely can't claim nvim saves me time given how much of it I spend tracking nvim related stuff 🤦🏿‍♂️

[vhyrro]

Update: a lot of problems have since fixed themselves. Well, not exactly "by themselves" lol, but treesitter in Neovim has had lots of updates which have drastically increased performance. A large problem used to be that injections would be entirely reparsed on every change, but the PR that made this incremental has been merged a while back and has dramatically increased speeds.

Changes have been made to Neorg core too - we rely very little on vim.schedules, which don't help much nowadays as much as they used to (although I can't exactly tell you why). The concealer also runs on fewer occasions now (only when you first read the buffer, not every time you enter one). This plus all the other small changes not worth mentioning here add up nicely!

To top it all off, the second generation norg parser is almost complete and ready to ship with mainline Neorg, and boasts higher parsing speeds as well as a lower memory/temporary allocation footprint. Closing this PR since it's no longer necessary :)