packer.nvim
packer.nvim copied to clipboard
wbthomason
Replace hererocks with native Lua implementation
This is a bit out there but could be quite useful, so I thought I'd open an issue to track/collect ideas.
The drawback of hererocks is that it adds a Python dependency -- with everything that entails (environment hell; see the ongoing issue with macOS). And since packer runs in Neovim, we are guaranteed to have a Lua interpreter (which is usually not the case when bootstrapping Lua) and may even assume LuaJIT. The main question is whether we can use Neovim's internal functions (or LuaJIT's os module) to replace Python stdlib functions.
I think we could replicate the small amount of functionality we actually need (download from a URL using curl or wget, extract an archive, run a build command, generate/otherwise have a sourcing script to set up paths).
If anybody is interested in taking this on, please, go for it! I can provide guidance on how to do this in packer's jobs/async framework.
I was going to open an issue when I saw this. From the chat it seemed unlikely to gain traction but I wanted to have the arguments in a linkable way. your last message makes me a bit more enthusiastic :) I find it a bit sad to require python in an otherwise full luarocks out there.
Wouldn't it be possible to shell out to luarocks ? like packer could install luarocks plugins in its own tree via luarocks install --tree ~/.local/packer/luarocks_packages <LUA PACKAGES>.
The point is how to get Luarocks in place (in the right version!) in the first place...
similar to python, simpler is to leverage the package manager. Packer can check ia luarocks is available for lua5.1 and ask the user to install it. For instance the current python infra fails for me because I dont have pthon but there is no feedback. I dont think packer should install either python or luarocks, just leverage the chosen solution when avaiable, otherwise asks the user. If packer gets packaged by distribs, it can be packaged with luarocks.
But it's not trivial to install luarocks with exactly the right settings (especially on some platforms) -- that's the whole point of using hererocks to ensure a valid install (otherwise we're basically back at python venv hell...)
Of course I don't have to tell you that, but Lua isn't built for a system-wide package ecosystem -- it's meant to be a self-contained embedded system. And fighting against that (which luarocks already does to some extent) will only end in tears...
@teto: I'm happy to add something to check for a system-level luarocks for Lua 5.1 already being installed and only use hererocks if it doesn't exist, but I do not want packer to have to check for the existence of a system luarocks package. That would require knowing how to query N different system package managers (pacman, apt, brew, nix, chocolatey, etc. etc.) and seems out of scope for packer.
I think a better discussion (which is a bit out of place here, however) to be had is whether neovim should not bundle luarocks as well (since it's already a build-time dependency).
(Even better would be to embed luarocks, but that is not yet supported.)
@teto: Oh, ok - sorry, but I think I'm confused, then. I thought you were arguing for checking for a luarocks package (i.e. not just checking for the presence of an appropriate luarocks binary already on PATH).
Regardless:
- Embedding Luarocks would be great but seems impossible unless Luarocks itself makes changes
- If Neovim bundles Luarocks, then great - the
packerLuarocks functionality can be trimmed back to just be "Invoke Luarocks to make sure that the rocks your plugin spec wants are installed/cleaned if no longer specified" - Otherwise, implementing the minimal chunk of
hererocksthat we need in Lua seems feasible - As a simpler partial solution, we can add a check for an existing Luarocks and only run the
hererocksinstall process if we have to
What's the complexity of shipping luarocks? IIUC these four bindeps needs to be provided https://github.com/luarocks/luarocks/tree/master/binary, otherwise luarocks is "just" lua files to bundle.
@bfredl Would it not be possible to just build Luarocks build normally during Neovim build (iirc something like this already happens?), then ship the two binaries (luarocks and luarocks-admin) and bundled Lua files that seem to comprise it (based on looking at the files provided by the Arch luarocks package)?
I dont think we should bundle luarocks (it's huge, not in line with distrib policies etc), yet it should be easy enough for plugin managers to shell out to it.
I've been pushing rockspecs to several plugin repositories (and to luarocks https://luarocks.org/labels/neovim) and so on nixos I am able to manage lua plugin dependencies via their respective rockspecs (no need to fiddle with it anymore). Nix being too confidential to drive the change, packer could be the force of change.
What I am looking for is a way for packer to detect a rockspec in a neovim plugin and automatically use it to detect the dependency. Is that poossible already ?
Such that use { 'lewis6991/gitsigns.nvim' } would automatically find and use https://github.com/lewis6991/gitsigns.nvim/blob/main/gitsigns.nvim-scm-1.rockspec to load dependencies ?
@teto While it's easy for packer to shell out to luarocks if it's installed, we still need to make sure (whether by the current approach or by simply requiring the user to install it themselves) that it is actually present (and compatible with the right version of Lua) on the user's system.
We can't currently detect and use a rockspec, but there's no reason for this other than I haven't implemented it yet. That would be a good feature to add; we could either check for a rockspec in the top-level directory of each plugin by default or only if a given keyword is present (e.g. rocks = true in the plugin spec). The biggest complication would probably be restructuring packer's job queueing a bit to allow us to batch new rock installs, which isn't too hard.
Agreed on the first part.
I've forked luarocks to generate nix expressions, there is some code to find rockspecs in a repo (but it uses luarocks API so may not be that helpful: https://github.com/nix-community/luarocks-nix/blob/test-speedup/src/luarocks/cmd/nix.lua#L386 A first approach could be to just take one random .rockspec in the rootfolder)
Yeah, I think it's reasonable to just take the root .rockspec, maybe preferring one with the same name as the plugin if one exists.
luarocks is actually sensitive to the name of the rockspec: it aborts if the name of the rockspec doesn't match the luarocks package name. But not sure how packer could get the package name: the repo name should work 90% of the time but that may not be enough. In general there should be just one rockspec in the repo so it should be straightforward. I plan to write a blogpost that I can share to explain the rockspec thing + plugin dependency, I hope this WE.
just for information, I've also seen several projects create a "rockspec" folder where they store the different rockspec for each version (and the dev rockspec at the root). There are the easiest project to work with but still a minority (see https://github.com/alerque/fluent-lua for one example).
what is the situation now? luarocks + hererocks constantly failing on macos, although there fix for brew, but none for nix.
I've reached what I think is a good design for nix (previous one was completely hackish) so I can finally merge soon. Basically I use ``lua_modules_path = "lua"` in the luarocks config and because there is no way to to tell luarocks where to install the "data" part of the rockspec I move it in a postinstall step. I've contributed a few rockspecs to bootstrap the project but having packer support it could help. Resorting to luarocks may not be perfect yet but with the help of the neovim community, we can fill the gaps and this could be a good solution.
Since 2 weeks ago, nixpkgs now can automatically package vim plugins and set their dependencies via the rockspec when it exists. I myself have been adding .rockspec to some key plugins (plenary.nvim, lush.nvim, nvim-cmp etc) but that's useless if plugin authors dont take over. For it to be useful, the plugins should also be advertised on luarocks.org and I dont like doing it myself (not scalable either). Nixpkgs is still niche but if packer could do something similar then surely plugin authors would follow. I've written some of my feedback here https://teto.github.io/posts/2022-06-22-neovim-plugin-luarocks-2.html (the writing was a bit rushed but I hope it's still understanble).
I will dump https://github.com/teto/vim2rockspec here as well, a simple script to generate a rockspec from a repo in case you wanna have fun
