uv icon indicating copy to clipboard operation
uv copied to clipboard

Support including dependency executables (e.g., `--include-deps`) in `uv tool install`

Open hynek opened this issue 1 year ago • 7 comments

hi, congrats on 0.3.0!

I've noticed one very useful pipx option that uv tool is currently missing: --install-deps. It means that it also adds the CLIs of packages it depends on. This is crucial for something like Ansible where the essential commands are hidden in sub-packages.

hynek avatar Aug 21 '24 09:08 hynek

It would be great if this could allow multiple dependencies to be installed in the same venv and for all of their executables to be exposed. More info in this rye feature request. This would allow me to install ansible,ansible-core and ansible-lint in the same venv, saving disk space and negating version mismatch issues.

ZingyAwesome avatar Aug 21 '24 12:08 ZingyAwesome

I've been hesitant to provide a flag to install all executables from all dependencies — what about --executable <name> to include executables from dependencies?

zanieb avatar Aug 21 '24 12:08 zanieb

The way it works in rye is that --include-dep names the particular dependency and installs then the tools that that particular dependency provides so it's not quite as crazy as installing all.

mitsuhiko avatar Aug 21 '24 12:08 mitsuhiko

Yeah that was what I meant, as long as you can specify multiple dependencies that way. My main issue with rye was that it allowed you to only specify 1 dependency to include executables from (or at least I couldn't get it to work with more than 1).

ZingyAwesome avatar Aug 21 '24 12:08 ZingyAwesome

I thought --include-dep can be provided more than once, but maybe there is a bug.

mitsuhiko avatar Aug 21 '24 12:08 mitsuhiko

I could perfectly live with uv tool install ansible --include-dep=ansible-playbook --include-dep=ansible --include-dep=ansible-inventory

Or however you call it; there's already the --with prefix used, so that might make sense too.

hynek avatar Aug 21 '24 12:08 hynek

This for ansible's install workflow is the last piece of me replacing pipx with uv tool so supporting this would be excellent.

jamesharris-garmin avatar Sep 09 '24 14:09 jamesharris-garmin

I had a look at all the linked issues and also asked some feedback out of band; I sketched some code to implement this, and I want to recap the proposed behavior. I'm not interested in debating the specific flag name right now, so let me call it --i-want-ponies for the moment.

Using ansible as an example, I'll highlight these three packages:

  • ansible: top-level package tool, provides the ansible-community executable and depends on ansible-core.
  • ansible-core: mainly exists as a dependency of the above, provides 11 executables (including for example ansible-playbook and ansible-inventory).
  • ansible-lint: sits on its own and provides the ansible-lint executable.

uv can offer the following:

  • uv tool install ansible: installs ansible-community.
  • uv tool install --i-want-ponies ansible-core ansible: like the previous, plus it installs all the 11 executables from its -core dependency (including for example ansible-playbook and ansible-inventory).
  • uv tool install --i-want-ponies ansible-core --i-want-ponies ansible-lint ansible: like the previous, plus it brings in an extra package ansible-lint and installs its executable. It doesn't need an explicit additional --with if the package is not already a dependency of the top tool.

As a side note, at some point this may end up bringing in too many executables (e.g. from fat packages like ansible-core), so possibly there will be a need to grow an additional flag to specify an allow-list of globs to filter executables by name. I'll leave that for a separate ticket.

lucab avatar Sep 20 '24 16:09 lucab

For the record as a workaround you can do this today

$ uv tool install ansible-core --with ansible

because ansible-core exposes the binaries, and the ansible package will provide all the plugins.

underyx avatar Sep 21 '24 18:09 underyx

This is a nice fix for the simple ansible case, but if my goal is to use ansible-lint I think I would have to install the tools as follows:

$ uv tool install ansible-lint --with ansible

In one invocation, then in a separate tool install:

$ uv tool install ansible-core --with ansible

which would mean that ansible-lint and the rest of ansible could get out of sync, which seems like it would be a nightmare if the two tool environments ever fell out of sync.

jamesharris-garmin avatar Sep 23 '24 19:09 jamesharris-garmin

same, plus I use community.aws.s3_cors module and currently forced to have 2 separate envs:

uv tool install ansible-core --with ansible --with boto3
uv tool install ansible-lint --with ansible --with boto3

in any case thank you so much for uv it's the GOAT

meoyawn avatar Sep 27 '24 18:09 meoyawn

@lucab This formulation sounds reasonable I would leave it up to developers of their tools to provide additional instructions. on how to install properly so that the user experience is correct. I could really use this approach to eliminate sync issues.

jamesharris-garmin avatar Oct 02 '24 15:10 jamesharris-garmin

Came here for the same problem with ansible-lint, that now has it's own environment. I also have some smaller projects with the same problem.

KalleDK avatar Oct 14 '24 08:10 KalleDK

While the workarounds work. It's very painful to have to go through every package we are installing to figure out which deps it has to then add a massive list of flags for each one.

This should be a simple flag "--include-deps" like pipx does.

I don't to install lets say "app A" , and then spend 1hr debugging why apps are missing cause I didnt include a bunch of extra "--include-deps" for each one.

gaby avatar Oct 17 '24 12:10 gaby

@gaby: out of curiosity, how many packages do you have with that problem? Of the 15 I'm installing, I only have the missing-executable-problem with ansible, to my surprise. What I'm installing is mostly individual tools like "ruff", "docutils", which apparently don't have dependencies with executables.

If you have a "tool-collection-of-my-company" package which groups useful tools as dependencies: yeah, that'll quickly become a pain to install currently.

(I'm all in favour of --include-deps, btw.)

reinout avatar Oct 17 '24 13:10 reinout

I don't to install lets say "app A" , and then spend 1hr debugging why apps are missing cause I didnt include a bunch of extra "--include-deps" for each one.

It seems way to easy to pull in a bunch of random executables this way? It sounds like what you need is some sort of uv tool show <name> --include-deps so you can just see the executables up front.

zanieb avatar Oct 17 '24 13:10 zanieb

@reinout Yes, besides ansible, pylint, ruff, uv, jc, gitlab (which has sdk/cli), the main use case is internal enterprise tools. Most of them are Typer or Textual apps that are bundled/distributed together in one or many wheels.

@zanieb You mean like before installing? Or if a tool is installed already to be able to see which apps it provides?

Right now we install things using pipx:

pipx install --global MyEnterpriseApp --include-deps. This install the app in a Global VENV (/opt/pipx/venvs/myenterpriseapp) and also adds each app cli to /usr/local/bin/.

These are multi-user systems were most users dont have sudo, they just use the provided tools by the system (python typer apps).

gaby avatar Oct 17 '24 13:10 gaby

maybe instead of (or as an alternative to) --include-deps, there should be

  1. an easy way to see all executables provided by the deps
  2. a CLI option to include executables from specific deps

E.g. I’d like to install jupyter, jupyter-lab, jupyter-execute and so on, all into one environment.

flying-sheep avatar Oct 24 '24 09:10 flying-sheep

We can also do... like uv tool install <package> --executable <name> --executable <name> and we can tell you what packages provide them or just allow installation from dependencies automatically at that point.

zanieb avatar Oct 24 '24 12:10 zanieb

That would be more guessable, and would allow for advanced things like like --executable 'jupyter-*'

I’d still also like (a flag for) uv tool list to show all the binaries that aren’t currently active.

flying-sheep avatar Oct 24 '24 16:10 flying-sheep

It seems way to easy to pull in a bunch of random executables this way? It sounds like what you need is some sort of uv tool show <name> --include-deps so you can just see the executables up front.

Could you explain the risk you are concerned about? With pipx if you use the --include-deps argument, you are specifically asking to install the executables from dependencies, which I wouldn't describe as installing "random executables".

Perhaps including a --dry-run option would help to ease concerns? e.g. uv tool install --include-deps --dry-run ansible. You already have the dry-run feature available in: uv pip install --dry-run. This could be extended to show the executables installed by each package when called with uv tool install.

stuartmaxwell avatar Nov 27 '24 19:11 stuartmaxwell

Could you explain the risk you are concerned about? With pipx if you use the --include-deps argument, you are specifically asking to install the executables from dependencies, which I wouldn't describe as installing "random executables".

Perhaps random was not a great word choice. There can be arbitrary transitive dependencies. It worry about it (1) being surprising which executables are installed and (2) that these executables could conflict with other tools.

A --dry-run option is a solution. I prefer a more explicit interface over --include-deps though.

zanieb avatar Nov 27 '24 20:11 zanieb

Perhaps random was not a great word choice. There can be arbitrary transitive dependencies. It worry about it (1) being surprising which executables are installed and (2) that these executables could conflict with other tools.

I think I still disagree with the term "arbitrary". If I use the --include-deps argument, then I'm specifically making an informed decision that I want all of the dependent executables installed too, there's nothing arbitrary about it. Sure, I might be surprised that Ansible includes the "ansible-doc" executable, which I've never used or needed, but I'm fine it being there in case I ever want to use it.

With regards to conflicting executables - isn't this only an issue when the executables are linked from the bin directory? And uv already handles the case where an executable already exists in the bin directory.

stuartmaxwell avatar Nov 27 '24 21:11 stuartmaxwell

Let's talk specifics. With pipx I like to install ansible and various ansible-related tools (ansible-lint, molecule) in the same virtualenv, which I do with

pipx install --incude-deps ansible
pipx inject ansible dnspython  # some of ansible plugins I use depends on dnspython
pipx inject --include-apps ansible ansible-lint
pipx inject --include-apps ansible molecule

(incidentally, I'd love it if uv tool supported inject. pipx remembers all the packages I've injected in a little .json file somewhere in the virtualenv and can reinstall them all at a later time with pipx reinstall-all, which is very helpful when I upgrade my system python from, say 3.12 to 3.13).

The thing here is that pipx inject has three modes:

  • pipx inject with no flags installs the extra packages but doesn't symlink any binaries into ~/.local/bin
  • pipx inject --include-apps installs the packages and symlinks their scripts, but doesn't symlink any scripts from transitive dependencies
  • pipx inject --include-deps installs packages and symlinks everything

If I'd used pipx inject --include-deps ansible-lint, then I'd get an extra, unwanted ~/.local/bin/black, since black is one of the dependencies for ansible-lint. I don't want my generic Python code formatter to be managed as part of an ansible-related tool virtualenv; I want to install and upgrade it separately and explicitly.

I guess this is the concern that @zanieb had: installing scripts from all transitive dependencies is sometimes undesireable. I found that pipx's solution (a separate pipx inject command) allows me to control this in a fine-grained way, and I'd be happy if uv adopted the same solution (uv tool inject [--include-apps|--include-deps] ...).

mgedmin avatar Nov 30 '24 17:11 mgedmin

uv tool install ansible installs ansible-lint, ansible-playbook, etc. but only symlinks ansible-community. The below one-liner creates these symlinks:

[ -d "$(uv tool dir)/ansible/bin/" ] && find "$(uv tool dir)/ansible/bin/" -mindepth 1 -maxdepth 1 -type f -executable -regextype posix-extended -regex '^((.+/)?)[^.]+' -print0 | xargs -0 ln -s -t "${HOME}/.local/bin/"

SVHawk13 avatar Dec 01 '24 22:12 SVHawk13

For whatever it's worth as another point of reference, my pipx clone in Zsh which wraps uv optionally takes a --cmd argument whose value is a comma separated list of commands to install. If that's not provided, and there's not an obvious single command from the whole installation (including dependencies), an interactive fuzzy finder in muli-select mode is invoked.

AndydeCleyre avatar Dec 01 '24 23:12 AndydeCleyre

I wanted to expand on the "enterprise" use case that gaby and reinout mention above (which I also have), since this thread has a lot of ansible examples, but I think this has at least one or two more use cases beyond that.

In my org, we ship an internal CLI application, that declares an optional extra bins, which is a collection of other CLI tools that are frequently used at the company. It's nice to be able to pipx install internal-cli[bins] --include-deps, and get not only our org-internal CLI, but these other useful tools available on the $PATH in one shot.

One particularly nice thing about this approach is that as new versions of the internal CLI are published with updates to the bins optional extras dependency list, people only need to pipx upgrade internal-cli to get these new tools which can easily be aliased into a self update command in our org CLI. self-update would be harder to implement if we needed to use a separate pipx install for every extra bin tool or something like --executable to explicitly list everything, though it's certainly not impossible!

Anyway, I hope that motivates a use case beyond "making ansible + uvx cleaner", but totally understand if you don't want to support this :-)

justin-yan avatar Dec 29 '24 04:12 justin-yan

My use case is that I want to install jupyterlab as a tool but have access to the jupyter executable, which I used to get with pipx install jupyterlab --include-deps.

It seems that the suggested uv tool install jupyterlab --executable jupyter would be a good solution to this. Is there a workaround to this for now?

devmcp avatar Jan 10 '25 15:01 devmcp

I have another idea: a new "uv tool link" command to link other files (like ansible-playbook).

uv tool link ansible-playbook just do "ln -s .local/share/uv/tools/ansible/bin/ansible-playbook .local/bin/ansible-playbook"

lgxz avatar Jan 27 '25 14:01 lgxz

I have another idea: a new "uv tool link" command to link other files (like ansible-playbook).

It's not a bad idea per se, but it means the user would have to know beforehand the names of all the executables. And some packages (e.g. Ansible) have many!

cu avatar Jan 28 '25 01:01 cu