minimalistic plugin loader to capture / handle errors

[Nowaaru]

⚠️ Please verify that this feature request has NOT been suggested before.

• [x] I checked and didn't find a similar feature request

🏷️ Feature Type

Other

🔖 Feature description

If any part of the NeoVim configuration errors, other well-functioning plugins will not loading as well. 100/10 times, this is problematic for debugging configurations or even simply trying to push through for five minutes to finish the last bit of work under a time crunch.

✔️ Solution

I propose a super-light coroutine-based plugin loading system. That is all.

❓ Alternatives

A pcall-based plugin-loading system. I think this is the worst option.

📝 Additional Context

No response

[horriblename]

what do you mean by coroutine-based? I have never seen anyone use it this context

one way I can think of is converting everything to lz.n managed. This adds overhead to plugins that don't need/can't be lazy-loaded but probably negligible.

[NotAShelf]

I've seen something like

local status_ok, ibl = pcall(require, 'ibl')
if not status_ok then
	return
end

used in the past. Perhaps something similar could be used to defer faulty plugins by ignoring setup. Although this only makes sense to mitigate loading path related issues, not setup tables. I don't think Lua/Neovim has a way of actually handling those. Do you know of a plugin manager or a configuration that has achieved something similar to what you have described?

[Nowaaru]

@horriblename what do you mean by coroutine-based? I have never seen anyone use it this context

one way I can think of is converting everything to lz.n managed. This adds overhead to plugins that don't need/can't be lazy-loaded but probably negligible.

When coroutines throw in Lua, they don't kill the main "thread" that is running the coroutine, similarly to pcall:

local thisWillError, res = coroutine.resume(coroutine.create(function()
  error("hi!")
end))

print(thisWillError, res)
--[[ Output:
false	Main.lua:2: hi!
]]--

If the coroutine succeeds:

local thisWillError, res = coroutine.resume(coroutine.create(function()
  return "ooga booga"
end))

print(thisWillError, res)
--[[ Output:
true	ooga booga
--]]

I say the coroutine approach better because of the ability to check the state of coroutines and act in several different ways with the result of coroutine.status, plus the ability to pass state through coroutine.yield and coroutine.resume. Though, the largest benefit coroutines have over pcall, however, is the ability for coroutines to not choke the main thread like pcall. Though, the overhead for coroutine construction is more observable with extremely large plugin configurations compared to pcall:

--- Lua 5.4
local a = os.clock();
local thisWillError, res = coroutine.resume(coroutine.create(function()
  return "ooga booga"
end))
local b = os.clock();
pcall(function()
  return "ooga booga"
end)
local c = os.clock();
local btoa = b - a;
local ctob = c - b;
print("coroutine time:", btoa);
print("pcall time:", ctob, ctob < btoa)
--[[ Output:
coroutine time:	5e-06
pcall time:	9.9999999999992e-07	true
--]]

@NotAShelf I've seen something like

local status_ok, ibl = pcall(require, 'ibl') if not status_ok then return end used in the past. Perhaps something similar could be used to defer faulty plugins by ignoring setup. Although this only makes sense to mitigate loading path related issues, not setup tables. I don't think Lua/Neovim has a way of actually handling those. Do you know of a plugin manager or a configuration that has achieved something similar to what you have described?

I don't think I know of any, but after a light search, it seems that lazy.nvim does this, yes.

[horriblename]

the largest benefit coroutines have over pcall, however, is the ability for coroutines to not choke the main thread like pcall.

you don't get free performance from using coroutines over pcall. coroutine would only make sense here performance-wise if 1. we have an IO heavy operation and 2. the plugin (assuming we're not doing the IO stuff) exposes a coroutine API for it. No plugin offers a coroutine based setup function so I highly doubt coroutines are even worth considering

as far as I can tell lazy.nvim only uses coroutines for async downloads/calling external commands and not for calling setup code of plugins

---

We could try to manually wrap each plugin config with pcall. Would be a heck lot of work and error prone

"Auto" wrapping each DAG section with pcall is probably not possible, since we quite often use variables defined in another section

[Nowaaru]

@horriblename

the largest benefit coroutines have over pcall, however, is the ability for coroutines to not choke the main thread like pcall.

you don't get free performance from using coroutines over pcall. coroutine would only make sense here performance-wise if 1. we have an IO heavy operation and 2. the plugin (assuming we're not doing the IO stuff) exposes a coroutine API for it. No plugin offers a coroutine based setup function so I highly doubt coroutines are even worth considering

Not understanding what you mean by "free performance" from using coroutines over pcall. I said that there is overhead from coroutine creation compared to pcall. By "choking" the main thread, I mean that it's a blocking function.

@horriblename No plugin offers a coroutine based setup function so I highly doubt coroutines are even worth considering

Why must a plugin exposing a coroutine-based setup function be necessary? I'm offering the use of coroutines in order to check the state of threads and report error messages into the log. Lazy does this already. If I'm not being short-sighted every use case of pcall - if this were to be implemented - should be just as valid if replaced with coroutines.

@horriblename "Auto" wrapping each DAG section with pcall is probably not possible, since we quite often use variables defined in another section

I don't see why this isn't possible regardless. You can modify variables beyond the scope of the thread (or function) with both pcall and coroutine. Is this not the case in NeoVim?

If anything, with pcall, it would be much more comfortable since variable assignment would be more predictable as it is blocking. Though, this shouldn't be a concern regardless because coroutines can "export" values both on yield and on coroutine completion: these exported values can be referenced in a _G-like table.

[horriblename]

The link you sent is lazy.nvim using coroutines for a build script. This allows their build script to run slow external commands without blocking. I fail to see how this is useful in our case, where we're mostly just running CPU-bound code to set up plugins. If it's error reporting you want, how would coroutine be better than just vim.notify() the error from a pcall?

I don't see why this isn't possible regardless. You can modify variables beyond the scope of the thread (or function) with both pcall and coroutine. Is this not the case in NeoVim?

yes we can use _G for sharing variables, what I meant by "auto" is "with no changes to existing plugin setup code", I should've made that clear, sorry

(note that some variables shared between DAG sections are local, though some of them are already global as well)

[Nowaaru]

what I meant by "auto" is "with no changes to existing plugin setup code"

Ouch. I get what you mean.

For what it's worth, I currently see no fault in just using the values returned from coroutine.resume in an "exports" table and passing that to dependents, completely removing the need for the use of _G entirely as coroutine.resume has arguments to be passed to the callee. The most change I can think of would be more related to whatever writes to init.lua (which hopefully isn't the same as the require("plugin").setup({...}) code itself) or maybe the deletion of dependency variable assignments in favor of function parameters.

If it's error reporting you want, how would coroutine be better than just vim.notify() the error from a pcall?

It's not error reporting in particular that's important, nor is the difference between pcall or coroutines, it's the fact that plugin setup can error out the main function. This consequently prevents unrelated plugins from loading entirely. The difference between the two functions to be used only affects scalability and maybe performance.

The two factors that nvf could benefit from are only the concurrency and the ability to run functions without unnecessarily blocking the main thread which I believe can lead to net performance gain even in the worst cases. The former factor is why I mentioned the DAG, because it makes race conditions a non-problem.

sorry for the back-to-back essays oh heavens

[horriblename]

ok, so implementation aside, what you want is a system that

• runs plugin setup code (DAG sections) in the order given by the DAG we have in nix
• when a a DAG section errors out, all sections depending on that one are skipped
• DAG sections that don't depend on the failed section run as usual

is that all?

we could write a loader like that but that looks a lot like what lz.n can do. In which case, I'd rather just try with lz.n instead. We'll revisit a custom solution if that doesn't work out (passing shared variables around sounds like a pain in lz.n)

The difference between the two functions to be used only affects scalability and maybe performance.

I don't have a clear picture of how our hypothetical custom loader would look like, so I don't really know how coroutines would help with scalability, for now I'll just hold on to the idea

[NotAShelf]

I'll write a more detailed response later. Two things for now:

1. We can fork lz.n, nixcats does it (?) and it can't be too hard. It would akso be nice to have our own loader implementation for the futute. It is not very large, surely it wouldn't be difficult to maintain

2. The DAG results are sorted through Nix; if we want to resolve them with Lua then the resolution order should be

Raw DAGs -> some lua evaluator -> Nix to sort the DAGs and finalize init.lua that will he passed to mnw.

[Nowaaru]

ok, so implementation aside, what you want is a system that

• runs plugin setup code (DAG sections) in the order given by the DAG we have in nix
• when a a DAG section errors out, all sections depending on that one are skipped
• DAG sections that don't depend on the failed section run as usual

is that all?

Yep yep, that's all of it. It wouldn't be counter-intuitive to allow this for manual config entries as well, yes? The descriptions make the option seem as if it's meant to be self-contained which (ideally) means less problems if this were to be implemented for the options as well.

[NotAShelf]

I'm revisiting this to make sure we don't let it die. There is value in a home-made plugin-manager (or a fork, even) so that we can, e.g., add custom events like VeryLazy directly in the plugin manager. Though I do not have a good idea yet on how to approach.

Problem would be having a Lua evaluator resolving DAGs instead of Nix, which I'd argue is more qualified for doing so. Unfortunately I lack the time to mess with this with my nvf backlog, and I'm going to revisit this at a later date if I want to make progress. Please remind me if I do end up forgetting.