pip-tools

pip-tools copied to clipboard

What's the problem this feature will solve?

pipx is a new tool which supports "global" installation of user tools. pip-tools seems like an ideal candidate for such a global tool. However, currently if you install pip-tools that way then anything that it installs (via pip-sync) goes into the pipx installation venv.

Describe the solution you'd like

I'd like to be able to install pip-tools once via pipx and then use this to sync my active virtualenv.

Alternative Solutions

The current alternative is to install pip-tools in each virtualenv but the downside is it then breaks pip freeze and requires updating for each virtualenv you might have on your system (pip install -U pip-tools).

I will try to tackle this myself soon, but I was wondering if someone already familiar with the pip interface could comment on what's going to need to change and possible stumbling blocks with doing this. Is it necessarily a huge change?

Hello @georgek,

Thanks for bringing this up! TBH I have no experience with pipx, but I've heard pretty much positive feedback about this project, so I think it would be cool to support it.

Internally pip-sync installs packages via:

subprocess.check_call([sys.executable, '-m', 'pip', 'install', '...'])

and it works pretty well for many cases. At first glance, we could configure the executable path that would resolve the issue. Any other ideas?

/cc @cs01 what do you think?

@georgek

but the downside is it then breaks pip freeze

Could you elaborate on why pip-tools would break pip freeze?

@atugushev

Could you elaborate on why pip-tools would break pip freeze?

It doesn't break it as such, but the problem is if you add tools like pip-tools to your virtualenvs then it will be included in pip freeze output which probably isn't expected. I see pip-tools in a similar class to tools like tox. They are developer tools and not part of a project runtime environment.

It looks like finding python via /usr/bin/env as is normal with scripts would do the right thing, but I haven't tested it yet. That's also platform specific so needs some care. Probably need to read the virtualenv docs to see about cross platform ways to detect a currently active virtualenv.

One reason I thought about this is because poetry supports (and recommends) installation via pipx. I think it's the way to go for tools like this.

One reason I thought about this is because poetry supports (and recommends) installation via pipx. I think it's the way to go for tools like this.

Yeah, that's because poetry creates and manages virtualenvs out of the box which makes things easy with pipx, but pip-tools doesn't.

Internally pip-sync installs packages via:

subprocess.check_call([sys.executable, '-m', 'pip', 'install', '...'])

Simply changing those calls to this seems to do the right thing:

subprocess.check_call(["python", '-m', 'pip', 'install', '...'])

This executes "python" in the currently active environment, which is what would generally be expected. I've tested it and it works fine when installed with pipx.

Calling sys.executable looks quite deliberate to me so I'm wondering if there are any downsides to changing it to simply "python"?

Being able to use pipx allows a much nicer workflow, IMO. You can install pip-tools once via pipx and easily keep it up to date using pipx. Then you can use pip-compile and pip-sync in any virtual environment without having to first install pip-tools. You'll always know you have pip-tools available thanks to pipx.

Calling sys.executable looks quite deliberate to me so I'm wondering if there are any downsides to changing it to simply "python"?

That was part of the fix #734 issue AFAIR. The python -m looks the problem solver, but could be also a breaking change though.

sys.executable guarantees that you will install packages in the right virtual environment.

It guarantees you'll install packages in the same environment as pip-tools is installed in but for me that's the wrong environment! I want to install packages in my currently active virtualenv regardless of where pip-tools is installed.

But one problem will be a mismatch with what pip-compile is doing. That imports pip which is coming from its own environment. I guess it would need to import from pip in the currently active virtualenv instead.

It guarantees you'll install packages in the same environment as pip-tools is installed in but for me that's the wrong environment! I want to install packages in my currently active virtualenv regardless of where pip-tools is installed.

This could probably solve this issue with pip-sync:

if os.environ.get('VIRTUAL_ENV'):
    executable = 'python'
else:
    executable = sys.executable

Hi just wanted to leave a quick note that I am aware of this issue, but haven't had time to fully read it yet and formulate a response. I am also a huge fan of pip-tools! Wonderful project.

I will respond at some point if my input is still needed. In the mean time I will cc some maintainers of pipx in case they have the bandwidth to respond. cc @uranusjr @itsayellow @gaborbernat

This could probably solve this issue with pip-sync:

if os.environ.get('VIRTUAL_ENV'):
    executable = 'python'
else:
    executable = sys.executable

This won't work well in some environments, like Devcontainer.

Devcontainer encloses the entire development environment into a container, per project, so there's no need to create new virtual environment and allowed to pollute system site-packages.

In that case, the code mentioned above won't solve the problem.

@georgek Does pip-sync --python-exectable $VIRTUAL_ENV/bin/python fix the issue?

Sorry for the extremely late reply. Yes, using the --python-executable option does seem to fix the issue with pip-sync, at least.

@georgek Did you make any progress on this feature, meanwhile?

I haven't looked into it recently. The --python-executable fix works for pip-sync, but not pip-compile. The former calls python in a subprocess, so can be made to work nicely (might just need UI changes).

But the latter (pip-compile) imports pip and does stuff with it (because it uses pip internals, not just exposed UI, for better or worse). I thought about creating a temporary virtualenv à la tox and doing the work in there. This would also mean it could support generating for various python versions.

I'd be happy to give this a go if this sounds in any way like a good idea. I have no experience in this area yet.

But the latter (pip-compile) imports pip and does stuff with it (because it uses pip internals, not just exposed UI, for better or worse). I thought about creating a temporary virtualenv à la tox and doing the work in there. This would also mean it could support generating for various python versions.

Yes, this sounds like the best solution. Unfortunately, there doesn't seem to be a way to import a module for a specific python version right now. Please try it out!

@georgek https://github.com/juftin/hatch-pip-compile was able to accomplish this with hatch. pip-compile isn't installed in the virtual environment of the target environment so the pip it used is a from somewhere else.

Perhaps @juftin can say more on how that was done especially being able to use the same pip-compile install across multiple Python versions (i.e., the hatch version matrix feature like in tox)

Perhaps @juftin can say more on how that was done especially being able to use the same pip-compile install across multiple Python versions (i.e., the hatch version matrix feature like in tox)

Actually, hatch-pip-compile does have to install pip-tools into its managed virtual environments to work properly because of this exact issue. hatch-pip-compile orchestrates running pip-compile from the inside the right environment using something like this:

subprocess.check_call([hatch_env.executable, '-m', 'piptools', 'compile', '...'])

Being able to use an option like --python-executable would be great to have and much simpler though.