[v3 Downloader] RFC: installing cogs/shared libs from PyPI + cog discovery

[calebj]

Feature request

Select the type of feature you are requesting:

• [x] Core (Downloader)

Describe your requested feature

Since redbot can be installed directly using pip, and will be usable without git once discord.py v1.0+ hits PyPI, I think that Red should support cogs or cog collections installed as python packages.

Fine-grained management via Downloader (beyond just calling [p]pipinstall) would be nice to have, but using setuptools plugin discovery (docs) is a good first step.

Benefits:

• Dependency management and updates are already handled
• Supports variants/dependencies for specific OSes or via setuptools "extras"
• Versioning, stability level and other metadata is well-defined
• Cog installation can be done in one step using pip install, without adding any repos
• Cogs can support pre-compiled native code packaged as a wheel
• Release pipelines are easier to set up

Current issues:

1. Package discovery for normal packages is independent of repo contents
   • packages not installed will not appear during discovery
   • sources of cog list/metadata could be a "base" package, or cogs.red.
   • base package would have to be updated when cogs are added/removed, description changes, etc.
   • cogs.red would have to perform discovery (or just rescan when a webhook fires)
2. PyPI has no tag/classifier for Red (yet)
3. Hybrid repositories
   • setup.py reading from info.json?
   • repo manager should tolerate namespace packages in git repos
   • installation may need to parse requirements.txt instead of JSON requirements array

[mikeshardmind]

Definitely interested in this being supported. Would make distributing my cogs with proper versioning much simpler. Would this also include better interaction between the downloader and python venvs? (I Should probably collect the list of odd behavior's I've noticed with venv usage mixed with current downloader using --target usage if not to ensure they are intended)

[calebj]

Would this also include better interaction between the downloader and python venvs?

Yeah. The idea is to install the cog package like any other pip package, which would automatically pull in the requirements for it. If it's in a venv, it would install to the venv's folder. Otherwise, it would put it downloader's lib folder. Loading it would import it from the regular packages folder instead of the installed cogs folder.

A "repo" in this case would be a list of references to packages on PyPI, and would pull the appropriate metadata from there.

I Should probably collect the list of odd behavior's I've noticed with venv usage mixed with current downloader using --target usage if not to ensure they are intended.

Please do. My WIP branch already disables --target if Red is running in a venv, virtualenv or pipenv, which should fix those issues if you're referring to the same ones I had. It'll also move the management of lib in sys.path away from Downloader.

[Proximyst]

I would just like to request the feature that is submodules to be allowed to be used in V3's downloader. I have already submitted a PR to this for V2, if you would like to check it out.

Thank you!

[calebj]

I'll add the submodule commands to what I've got working so far as per the PR. But it'll be tricky getting patch notes working for them, so those might be barebones/commit messages only until I have the rest stable and can focus on this.