Create a package index

[cweagans]

Vim users have vim.org as a package index to some extent, and then a bunch of random repos on Github, but I think we can do better.

I'd like to create something similar to Vim Awesome, but host it at packages.neovim.io or plugins.neovim.io. I think that while we could do this with Github Pages, it would be a better user experience to build it with something a little more robust (i.e. some server-side programming language). Obviously, this system would be open source, but we should provide the canonical version for Neovim.

I'm willing to build and host this system. I'd also like to suggest that we place one non-obtrusive ad (something like somewhere on the page. I'd propose a 30/70 split of the proceeds from that ad - 70% going to Neovim (ideally, to help pay for more full time work on Neovim), and 30% going to me to help recoup hosting costs (as I expect that the main instance could get fairly high traffic over time).

There would be no cost to users to use the service, and anyone would be able to set up an account and list their package. I imagine this working a bit like Packagist (for anyone familiar with the PHP world), Dockerhub (for Docker users), or npm (for Node.js users).

Basically, a user signs in with Github or Bitbucket. The application then gets a list of their repos, and they can click a button for a given repo to add that repo to the index. Depending on what the app can parse out of their repository (name of the plugin, description, dependencies, etc), there may be a second step to ask them to clarify any additional information. When they click go, we'll save the entry and add a webhook to that repository so that we're notified whenever there are new commits/tags so that we can display the appropriate information to the user.

One thing that's really important to making this service scale reasonably well is to not try to crawl through all of the Github repositories like Vim Awesome does. I think that's a massive waste of time and computing resources. It's much better to just have users add their packages and be notified when they're updated.

Since this service would also be open source, users would be able to spin up an instance of their own to host private packages if they wanted to (for instance, maybe a company has internal editor plugins that really only make sense to distribute to that company's employees). I imagine this process would look something like:

1. Install the software
2. Add a line to your vim config to include your package index
3. Use whatever package manager you use normally, and the packages included in your own index will be included in searches and whatnot.

This also opens the door to having real machine-readable dependency lists for a given plugin, and this applies to system config/supporting software, vim configuration, and what other plugins are installed. For instance, Syntastic depends on having a vim with "autocmd, eval, file_in_path, modify_fname, quickfix, reltime, and user_commands" enabled. A package manager that's aware of those platform dependencies can let the user know when they install the package that they're missing something important. clang_complete depends on having clang installed on the system. And Unite used to depend on vimproc, so a package manager that was aware of that relationship could have just automatically installed vimproc when Unite was installed.

@tarruda @justinmk What do you think about having something like this as the "official" neovim plugin list? I may have asked about this before, but I can't recall.

The value I see in this is that there's one place for people to go look for packages. Discoverability of vim packages has been a pretty difficult thing for me in the past, and it ends up being very time consuming. Having one index with search, categorization, and user ratings/reviews seems like the perfect way to resolve that issue, and having it open source means that the community can help to improve it over time.

[Shougo]

@cweagans

I and @tyru have considered the plugins repsitory system.

Our project is in: (Sorry, the discussion in Japanese) https://github.com/vim-vivacious/metainfo/issues/

[justinmk]

@cweagans Note https://github.com/neovim/neovim/issues/247.

What do you think about having something like this as the "official" neovim plugin list?

On paper it's great, but the details require thought and it will be a huge time-sink to get it right. IMO. I'm not at all opposed to someone taking total ownership of this and making it work, and I'll do anything I can to remove obstacles that are blocking progress. But taking an active role in this is low-priority until other more important problems are solved.

Designing a package format is the first challenge. Personally I am strongly in favor of the "passive" approach that VimAwesome takes--it avoids the need to design a package format, for the near-term. Any technical challenges with crawling repos and scraping vimrcs seem much more tenable to me than designing a package format or centralized repo and getting the community on-board.

As always I would strongly recommend taking a look at existing systems (VimAwesome) and trying to fix them instead of starting a new effort. Though VimAwesome's software stack seemed ridiculous when I glanced at it.

[cweagans]

@justinmk I don't want to design a package format. I just don't want to crawl Github. It's simple enough to login with Github and essentially say "my repo is a Neovim plugin". Rather then requiring a metadata file somewhere, we'll just ask the user fill in some details in the web UI. Later, if we decide we want to ask plugin maintainers to include a metadata file, it will be easy to just grab that file, pre-fill the web UI metadata fields, and make it so that the user doesn't have to see/manually edit them.

The only thing that the web UI would be for is a curated index of plugins that are known to work with Neovim. Then there would be some JSON output for plugin manager plugins to consume as a way to get a complete package listing (for search or for downloading a specific release of a plugin).

I've looked through the source of Vim Awesome and it is...well....not awesome. I wouldn't want to reuse that.

[tyru]

If neovim thankfully uses our idea of metainfo repository, please see the overview image. The image is totally written in English because we suppose a workflow which is a part of Vim ecosystem(vim-pi,...) in feature.

[justinmk]

@tyru what is the difference between vim-pi and vivacious?

[tyru]

Well, also I thought that like "well, vim-pi seems enough for me" until I talked with @Shougo.

According to his story,

1. Plugins that does not exist are listed in databases yet

Almost data are until 2014. Google Code is closed but vim-pi doesn't check if the plugins exist so the plugins are remaining.

2. Need to run a vim script to get metadata

It doesn't have a whole JSON in repository which package managers want. Need to run a vim script and generate a whole JSON. (In the overview image: "vim-pi.json") @Shougo created a snapshot whole JSON and uploaded to gist. his package manager now seems to use that. https://gist.github.com/Shougo/028d6ae320cc8f354f88

3. URL Protocols of plugins in databases are "git".

However, this problem can be solved by replacing the URLs with https one. I and @Shougo think that a package manager should not use a plain protocol like git, http.

4. vim.org is not https

This is the problem of vim.org . No https site of vim.org .

However, I thought that this problem can be solved by using vim-scripts repository. But @Shougo told me that vim-scripts.org is not working.

Well, I didn't ask about this then, @Shougo, Please describe why you thought that vim-scripts.org is not working?

[tyru]

The image is just an overview and we are just starting to work on that. So please tell me if the overview is ambiguous. And no codes are available yet... But if you thankfully help us to make it, we should output all information in English.

[Shougo]

Well, I didn't ask about this then, @Shougo, Please describe why you thought that vim-scripts.org is not working?

Because, the bot does not work...

https://github.com/vim-scripts

It should be updated everyday like vim-pi bot.

[tyru]

@Shougo I see. Indeed it seems a little bit old.

[cweagans]

@Shougo @tyru IMO, that's way too complicated. There's really no reason for all of that complexity if you look at what the requirements actually are:

1. We want a list of plugins that work with Neovim.
2. The list should be separated by category and searchable in a web UI
3. The list should be available in a machine readable format for package managers to consume.
4. There should be some guarantee that the list has not been tampered with via MITM or otherwise.
5. The list should include download locations for releases.

Crawling Github repos is not a valid approach in my opinion. It's just not worth the computing and network resources it requires, especially when you can re-architect things a bit and use a solution that doesn't require a crawler. Web hooks (from either Bitbucket or Github) go a long way toward eliminating the need for crawling, since those services will simply notify the packages site any time there is a change (inc. a new tag or something).

Personally, I'd like to encourage people to collaborate rather than creating new plugins for every little thing, too. That is, there might be the "official" PHP plugin, that provides solid autocomplete, syntax highlighting, filetype detection, etc, and rather than having some other plugin that adds docs lookup, they just add it to the official one. That's down the road though.

In any case, if we want to have an official packages site, I can build it and host it without any problem. @tarruda, @justinmk -- if you want to see a prototype before committing, let me know, and I'll throw something together for you to look at.

[justinmk]

It's just not worth the computing and network resources it requires,

Do you have numbers on that? I somehow doubt it is very expensive.

Web hooks (from either Bitbucket or Github) go a long way toward eliminating the need for crawling

But it's not equivalent. Passive always wins, if it can possibly work. The value in VimAwesome's approach (were it actually maintained) is that it shows actual usage of plugins by users, instead of just upvotes or stars.

if we want to have an official packages site, I can build it and host it without any problem.

We can link to it and suggest it, but declaring anything as "official" is a long way off, it incurs all kinds of expectations that the project isn't prepared to meet. I haven't even looked at vim-pi or vivacious :/

[tyru]

@cweagans

Crawling Github repos is not a valid approach

Yes. We also talked about that and the bot does not have to crawl GitHub repositories via GitHub API. We just provide a list of plugin URIs (see "URI List File"), and GitHub bot crawls only their repos in the list file.

Web hooks (from either Bitbucket or Github) go a long way toward eliminating the need for crawling

Hm, the idea sounds Vim plugin developers must specify the Web hooks URLs from GitHub or BitBucket site's "Settings" page. I don't think all Vim plugin developers (who have a repos in GitHub, BitBucket) easily folow to do that work.

Personally, I'd like to encourage people to collaborate rather than creating new plugins for every little thing, too

Yup, certainly our idea doesn't have the system like community hub site. It is just for developers, not users. Because I and @Shougo just supposed to suggest our idea as a part of Vim community's ecosystem to Vim community. We just talked about metadata repository of Vim plugins. not hub site. If you also think to have the system like "official" plugin and hub site, personally, I will not intend to implement a such system.

However, I don't know about Neovim's current ecosystem... ;) @Shougo seem to involve eagerly with Neovim, vim-pi :)

[cweagans]

the idea sounds Vim plugin developers must specify the Web hooks URLs from GitHub or BitBucket site's "Settings" page.

You'd log into the package index with your Github or Bitbucket account. The application would show a list of repos, and when you select one to add to the package index, the webhook would be set up for you automatically. This is similar to how Travis CI works.

The value in VimAwesome's approach (were it actually maintained) is that it shows actual usage of plugins by users, instead of just upvotes or stars.

This is easily doable. Packagist (the PHP equivalent to NPM) does this by tracking the number of downloads. Composer, the CLI client, is expected to send a request to a specific Packagist endpoint whenever a package is downloaded so that number of downloads can be tracked.

Do you have numbers on that? I somehow doubt it is very expensive.

Off the top of my head, this is what I can think of:

• You need to discover repos in the first place. This is pretty expensive because it requires using the search API from whatever VCS host you're interacting with (Github, Bitbucket, etc) to find the repos in the first place.
• Once you've gotten a list of these repos, you need to keep them in a queue which could get quite large.
• You need to be able to guarantee that a repo would be crawled at a specific interval (n hours) so that repo metadata stays up to date, which means that your entire queue must be able to be processed within n hours.
• As the list of plugins grows, you'll need to be able to add computing resources to keep up with the queue size to guarantee the previous point.
• As the Neovim community grows and diverges from the main Vim community, you'll also need to be able to somehow differentiate Vim plugins from Neovim plugins, which could mean scanning for specific keywords in the source code of the repo that is being crawled, which means that the crawler would need to be capable of doing a git clone.

None of this is particularly expensive at the beginning, but the expense grows over time. With the webhook approach, the cost only grows with the number of users actively using the web UI (the metadata requests can easily be cached on a CDN, so there's not really a scalability problem there).

We can link to it and suggest it, but declaring anything as "official" is a long way off, it incurs all kinds of expectations that the project isn't prepared to meet.

Understood. I think that Neovim should have (at least as a long term goal) an official package hub, as other communities do, as it really helps with discoverability of editor plugins. Even Emacs has this now. The upside to saying that it's an "official" thing is that Neovim itself could include a canned API to interact with this service (i.e. just some functions you can call from your own vim scripts to find, download, enable packages), so plugin managers would essentially become the UI layer on top of this API.

Even longer term, I'd love to see the Neovim community settle on one package manager and eventually move it into core, so that Neovim itself would ship with a way to manage plugins like Atom and Emacs. This is a very long way off, though. Just communicating my vision at this point.

[tarruda]

@cweagans If you want to do this, I suggest to investigate how component.io (javascript package registry and installer) is implemented. It is simple, well designed and has minimal server-side requirements, so it would be a good idea to learn from it(even though it is currently unmaintained). Here's what I could understand from a quick search through their repositories:

• component.io is a frontend for search.js which is a simple engine for filtering package descriptions contained in json files.
• search.js gets its json from the registry, which also functions as the crawler for github projects.
• crawler.js is a simple web app containing a single input box for github usernames. When a username is entered, the server will add projects containing a "component.json"(the package manifest) from that user to the package database(which is the json file queried by search.js).

I found this design interesting because it allows us to use github pages: the crawler can simply push new json files to the github page repository and they will be available on the web UI. A server is still required for running the crawler, but this is much simpler than maintaining a dynamic web app with github authentication(only used for registering/removing packages).

[cweagans]

Coming back to this:

I don't think we need to create a registry now. Github does a lot of the heaving lifting for us. That is, we don't need a separate registry at all because Github itself can act as a registry! All we need is a nice UI on top of the repository search API.

Essentially, the process for opting-in to including a plugin in the listing would be to tag your repo with "neovim-plugin" or similar (we can work out the specifics later). Then, a UI on neovim.io would allow a user to basically browse Github's API in a nice way.

Also, this only addresses discoverability of plugins, which is my primary interest. The actual installation, dependency resolution, etc would all still be handled by 3rd party plugin managers.

I'd be willing to prototype what I'm thinking if that would be of interest. Note that we could also use this UI to replace https://github.com/neovim/neovim/wiki/Related-projects.

[justinmk]

If one had to choose only one new thing, the single most valuable thing that could be done in this space would be to ship a way for users to install a plugin manager by a single command. Discovery would be the next thing, but currently for new users, setting up a plugin manager is where the friction is.

Yes, it's more boring than creating a new web app :)

[cweagans]

Fortunately, we don't only have to choose one :)

I don't disagree that installing a plugin manager with a single command would be awesome, but that's not really something that I'm well equipped to help with at the moment.

[justinmk]

see also:

• @teto exploration of Luarocks as the Neovim package manager: https://teto.github.io/posts/2021-09-17-neovim-plugin-luarocks.html
• https://neovim.discourse.group/t/plugin-metadata-formats-what-has-been-tried-already-why-did-they-fail-and-can-it-succeed/1153