wbond
Better dependency management - install only matching version and more ideas
Right now, Package Control installs a dependency for every platform and every ST version:
That's not really nice. Would be much better if a dependency is installed only for the matching platform/ Sublime version.
I think this is mostly a matter of developer time.
Would you be interested in helping create and update the 10 package files per dependency release?
Alternatively, do you have ideas for automating the creation of .sublime-package files and uploading them to the GitHub release functionality?
Yes, there is an API for it https://developer.github.com/v3/repos/releases/#upload-a-release-asset. However, I'm not sure if it's a good idea to lock into GitHub.
Removing non-matching directories after install would be a first start.
In the longer term I have a few other interesting ideas which should probably considered as a whole:
- install (non-binary) dependencies from PyPI
- install (binary) wheels on Windows and OSX
- try to compile packages on Linux if the toolset is available (as there are no wheels for Linux)
- version pinning and conflict resolution: install dependencies separate per Package so there are no conflicting dependencies (e.g. package A can require library 1.0 and package B can require library >=2.0). Related to #800
Those are some pretty ambitious ideas that will be very hard to integrate, if at all. While they would definitely be awesome to have, especially being able to use the pypi repository, the packages there are specifically packed to be used within their toolchain (e.g. setuptools) and Package Control would have to emulate them in order to make the packages go where they should.
I thought about a virtual env setup for packages for a bit but I am not sure how you would accomplish this considering every packages technically runs in the same environment. I do not know enough about Python loaders to be able to tell whether it's possible to yield different results for import statements depending on where they are issued, but it might very well be. Hopefully ST2 won't be a factor anymore in the near future.
I'll start by saying I personally don't have an interest in using, or developing those longer term goals. The reasons being:
- Lots of development time, which no company is going to sponsor
- Turning the Sublime Text Python environment into a big intertwined mess of required dependency versions that can't be resolved
- Limited use by package developers - heck most can't be bothered to create tags for releases
- There exist better solutions
Personally I think the one true approach is to embed pure Python packages from PyPi as submodules and use relative imports. Now each package owns its dependencies, and can upgrade, or monkey patch as needed. No pinning, conflict resolution or anything else needs to be written.
The only reason I developed "dependencies" the way I did was for the following reasons:
- Compiling shared libraries on all environments is a pain - there are 10 targets for ST2/3, how can we easily share the benefits of that work
- We should not have multiple packages trying to fix SSL on Linux
- Downloading 5+ MB of dependencies for each minor patch release of a package is a huge waste
- We already are doing this kind of thing, like with Emmet and PyV8
So from that perspective, pure Python package from PyPi are useful since we should be embedding them as submodules. Dealing with figuring out how to get a wheel installed now means we have to deal with dependency dependencies and setuptools. And trying to wrap different compilers, etc on Linux is a support nightmare.
That said, if there are developers who want to go down this path, I am not going to stop their efforts unless it significantly complicated the core functionality of Package Control. Ideally it would start with a discussion of ideas for the high level approach, and then transition into code and pull requests.
This isn't meant as a dig at anyone, or anything like that, but the old "dependencies" issue sat untouched with scant more than a handful of high-level comments and no code for over a year and a half. Unless there ends up being a resurgence of interest from some experienced Python devs with a copious amount of free time, I tend to think we will just see some slight iteration on the current functionality.
Personally, I doubt I'll have significant time to work on Package Control in the next 6-9 months. I just put in over two months, and am constantly doing little things related to maintenance of the code and server. Right now my attention is focused on getting back to paying the bills and supporting my family. :-)
Personally I think the one true approach is to embed pure Python packages from PyPi as submodules and use relative imports. Now each package owns its dependencies, and can upgrade, or monkey patch as needed. No pinning, conflict resolution or anything else needs to be written.
I recently took this approach and vendored dependencies for Requester, instead of using the Package Control dependencies feature. I want to share what I learned, get feedback from people who have been thinking about the problem, and maybe move the discussion on dependency management forward a bit.
Basically, I think Will is right. This is a better approach for a bunch of reasons, e.g. you can easily install dependencies of dependencies, instead of having to create a new repo for each sub-dependency and getting it merged into package_control_channel.
I made a little script to help me install/vendor a package's dependencies. It can be run from the root of the package:
# ensure `pip install --target ...` works
echo "[install]
prefix=" > setup.cfg
mkdir -p deps
touch deps/__init__.py
pip install --target "./deps" -r deps.txt
It assumes you have a file called deps.txt, with the same format as requirements.txt, and it installs all deps into a deps directory in the root of the package repo.
Imagine you have a package called Package. In package.py, you load all of the package's commands, which looks something like this:
from .add_path import add_path
with add_path(__file__, '..', 'deps'):
from .commands import *
add_path is a context manager that puts '../deps' at the front of your Python path.
It's crucial that your deps be in your Python path, even if your code does relative imports of deps, because your deps import their deps with absolute imports. If you don't do this, you have to go in and monkey-patch all of your deps to also perform relative imports, which in my opinion is not an acceptable solution.
This approach ensures your deps are available for import by your package's commands. All you do is use a relative import in a command module, e.g. in commands/my_commands.py you would do from ..deps import my_dep.
Using the deps.sh script means you can change easily upgrade or change these dependencies without worrying about package_control_channel. This is almost a complete solution, but there's something very important it doesn't handle: allowing different packages to depend on different versions of the same dep.
I've thought about this problem, and it seems much harder to solve. Because plugin_host is a single Python environment, it's basically impossible to simultaneously use two Python packages with the same name. The one that was imported first "wins".
So if you want to ensure your package uses a specific version of a dep that might also be vendored by another package, you have two options:
- Manually rename/namespace your deps. This requires monkey-patching a bunch of your deps' code, which makes it IMO an unacceptable solution.
- "Unimport" all your deps and reimport them every time one of your commands is run.
The second solution technically could work, but it's sort of impracticable, because you would need to reimport these deps in every one of your package commands' run methods. You couldn't just do this at the top of the command modules, because those top-level import statements are only run once, when the plugin is first loaded.
That's as far as I've got. Would love to hear what @wbond , @FichteFoll , @asfaltboy , or anyone else thinks. IMO better dependency management for packages is the most important improvement that can be made to ST, and I'd love to make some strides here before ST4 comes out =)
Optimally, each extension could run in it's own thread, clearing any "shared import cache" (sys.modules) on load, this should help with isolation and no dependency conflicts should occur; though, I'm not sure if we do this today or we use only 1 thread for all extensions.
Another solution we could consider is to bundle the venv (including all dependencies, the plugin package, alongside the platform specific python binary/ies) as a self contained zip in the style of pex. The difficulty here is the build step would require a CI with access to all target platforms.
Package Control supports installing python packages (libraries) from platform specific *.whl files.
Legacy dependencies are converted to python packages and moved to $DATA/Lib/ path, which removes all unneeded versions not matching current platform.
