pip-tools
pip-tools copied to clipboard
Support for installation via pipx
What's the problem this feature will solve?
pipx
is a new tool which supports "global" installation of user tools. pip-tools
seems like an ideal candidate for such a global tool. However, currently if you install pip-tools
that way then anything that it installs (via pip-sync
) goes into the pipx
installation venv.
Describe the solution you'd like
I'd like to be able to install pip-tools
once via pipx
and then use this to sync my active virtualenv.
Alternative Solutions
The current alternative is to install pip-tools
in each virtualenv but the downside is it then breaks pip freeze
and requires updating for each virtualenv you might have on your system (pip install -U pip-tools
).
I will try to tackle this myself soon, but I was wondering if someone already familiar with the pip
interface could comment on what's going to need to change and possible stumbling blocks with doing this. Is it necessarily a huge change?
Hello @georgek,
Thanks for bringing this up! TBH I have no experience with pipx
, but I've heard pretty much positive feedback about this project, so I think it would be cool to support it.
Internally pip-sync installs packages via:
subprocess.check_call([sys.executable, '-m', 'pip', 'install', '...'])
and it works pretty well for many cases. At first glance, we could configure the executable path that would resolve the issue. Any other ideas?
/cc @cs01 what do you think?
@georgek
but the downside is it then breaks
pip freeze
Could you elaborate on why pip-tools
would break pip freeze
?
@atugushev
Could you elaborate on why
pip-tools
would breakpip freeze
?
It doesn't break it as such, but the problem is if you add tools like pip-tools to your virtualenvs then it will be included in pip freeze
output which probably isn't expected. I see pip-tools
in a similar class to tools like tox
. They are developer tools and not part of a project runtime environment.
It looks like finding python via /usr/bin/env
as is normal with scripts would do the right thing, but I haven't tested it yet. That's also platform specific so needs some care. Probably need to read the virtualenv docs to see about cross platform ways to detect a currently active virtualenv.
One reason I thought about this is because poetry supports (and recommends) installation via pipx
. I think it's the way to go for tools like this.
One reason I thought about this is because poetry supports (and recommends) installation via
pipx
. I think it's the way to go for tools like this.
Yeah, that's because poetry
creates and manages virtualenvs out of the box which makes things easy with pipx
, but pip-tools
doesn't.
Internally pip-sync installs packages via:
subprocess.check_call([sys.executable, '-m', 'pip', 'install', '...'])
Simply changing those calls to this seems to do the right thing:
subprocess.check_call(["python", '-m', 'pip', 'install', '...'])
This executes "python" in the currently active environment, which is what would generally be expected. I've tested it and it works fine when installed with pipx
.
Calling sys.executable
looks quite deliberate to me so I'm wondering if there are any downsides to changing it to simply "python"
?
Being able to use pipx
allows a much nicer workflow, IMO. You can install pip-tools
once via pipx
and easily keep it up to date using pipx
. Then you can use pip-compile
and pip-sync
in any virtual environment without having to first install pip-tools
. You'll always know you have pip-tools
available thanks to pipx
.
Calling
sys.executable
looks quite deliberate to me so I'm wondering if there are any downsides to changing it to simply"python"
?
That was part of the fix #734 issue AFAIR. The python -m
looks the problem solver, but could be also a breaking change though.
sys.executable
guarantees that you will install packages in the right virtual environment.
It guarantees you'll install packages in the same environment as pip-tools
is installed in but for me that's the wrong environment! I want to install packages in my currently active virtualenv regardless of where pip-tools
is installed.
But one problem will be a mismatch with what pip-compile
is doing. That imports pip
which is coming from its own environment. I guess it would need to import from pip
in the currently active virtualenv instead.
It guarantees you'll install packages in the same environment as
pip-tools
is installed in but for me that's the wrong environment! I want to install packages in my currently active virtualenv regardless of wherepip-tools
is installed.
This could probably solve this issue with pip-sync
:
if os.environ.get('VIRTUAL_ENV'):
executable = 'python'
else:
executable = sys.executable
Hi just wanted to leave a quick note that I am aware of this issue, but haven't had time to fully read it yet and formulate a response. I am also a huge fan of pip-tools! Wonderful project.
I will respond at some point if my input is still needed. In the mean time I will cc some maintainers of pipx in case they have the bandwidth to respond. cc @uranusjr @itsayellow @gaborbernat
This could probably solve this issue with
pip-sync
:if os.environ.get('VIRTUAL_ENV'): executable = 'python' else: executable = sys.executable
This won't work well in some environments, like Devcontainer.
Devcontainer encloses the entire development environment into a container, per project, so there's no need to create new virtual environment and allowed to pollute system site-packages.
In that case, the code mentioned above won't solve the problem.
@georgek Does pip-sync --python-exectable $VIRTUAL_ENV/bin/python
fix the issue?
Sorry for the extremely late reply. Yes, using the --python-executable
option does seem to fix the issue with pip-sync, at least.
@georgek Did you make any progress on this feature, meanwhile?
I haven't looked into it recently. The --python-executable
fix works for pip-sync
, but not pip-compile
. The former calls python in a subprocess, so can be made to work nicely (might just need UI changes).
But the latter (pip-compile
) imports pip and does stuff with it (because it uses pip internals, not just exposed UI, for better or worse). I thought about creating a temporary virtualenv à la tox and doing the work in there. This would also mean it could support generating for various python versions.
I'd be happy to give this a go if this sounds in any way like a good idea. I have no experience in this area yet.
But the latter (
pip-compile
) imports pip and does stuff with it (because it uses pip internals, not just exposed UI, for better or worse). I thought about creating a temporary virtualenv à la tox and doing the work in there. This would also mean it could support generating for various python versions.
Yes, this sounds like the best solution. Unfortunately, there doesn't seem to be a way to import a module for a specific python version right now. Please try it out!
@georgek https://github.com/juftin/hatch-pip-compile was able to accomplish this with hatch. pip-compile
isn't installed in the virtual environment of the target environment so the pip it used is a from somewhere else.
Perhaps @juftin can say more on how that was done especially being able to use the same pip-compile install across multiple Python versions (i.e., the hatch version matrix feature like in tox)
Perhaps @juftin can say more on how that was done especially being able to use the same pip-compile install across multiple Python versions (i.e., the hatch version matrix feature like in tox)
Actually, hatch-pip-compile
does have to install pip-tools
into its managed virtual environments to work properly because of this exact issue. hatch-pip-compile
orchestrates running pip-compile
from the inside the right environment using something like this:
subprocess.check_call([hatch_env.executable, '-m', 'piptools', 'compile', '...'])
Being able to use an option like --python-executable
would be great to have and much simpler though.