hatch-pip-compile
hatch-pip-compile copied to clipboard
juftin
Hard-coded reliance on global `pip`? - Error: `/usr/bin/python: No module named pip`
The error is straightforward, but the reason for it is not. I want to use uv everywhere.
All I did was create a new test project with hatch new foo, then add the minimal config sections as well as the desired uv settings to the generated pyproject.toml. Here is an abbreviated version:
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "foo"
version = "0.0.1"
[tool.hatch.env]
requires = [
"hatch-pip-compile"
]
[tool.hatch.envs.default]
installer = "uv"
type = "pip-compile"
pip-compile-resolver = "uv"
pip-compile-installer = "uv"
To get that error I just ran any hatch command, e.g. hatch run ls or hatch env show.
I don't have pip installed globally on my system, nor do I want it. I want to use uv everywhere. It seems somewhere in the code of hatch-pip-compile it relies on the existence of a globally installed pip for some reason.
It also does not help to debug the problem that all I get is that line /usr/bin/python: No module named pip. No traceback or anything. Because of this, I did not even know where to begin to search in the source code for the cause of the error. It is surprising that the chosen Python executable also seems to be the system Python, rather than any of the Hatch-Python versions, but that may be a separate issue.
I would love to use this plugin because I think it fills a giant hole in Hatch's dependency management. Please let me know, if you need any more information or for me to try something specific.
PS: I just saw that pip-tools is a base dependency here, which in turn of course depends on pip. I don't know, if that is the cause of the issue. But if you want to allow people to use uv instead, as it seems from the documentation and the options to use it as a resolver and installer, that might not be a great idea.
The plugin shouldn't be using a global pip and instead using the virtual environment's pip - it does this by calling python -m pip ... inside an activated virtualenv.
pip-tools is actually a dependency regardless of being used as an installer/resolver because we use it to parse a requirements file and ensure every dependency is in sync: https://github.com/juftin/hatch-pip-compile/blob/e8ed56dc61d352fb3a6b592eb0891eb3bb7b2b28/hatch_pip_compile/lock.py#L16
I'll need to figure out how to replicate this so I can resolve it - do you know of any docker images I might be able to use to replicate your issue?
@juftin Thank you for the quick response. It seems to me (though this is just an educated guess) that it does not even get to the virtual environment. The error comes up immediately without noticeable delay. I'll see, if I can make this more easily reproducible with a docker image and get back to you.
So far the best I can offer is a few more system specs:
- Arch Linux
6.10.3-arch1-2 - Python
3.12.4 - Hatch
1.12.0
As a side note, I am not sure it is necessary to use pip-tools for the in-sync-check, but I have not looked into the code. Maybe I am misunderstanding something. Also using non-public modules seems risky, but this is neither here nor there.
That sounds good - some kind of Docker image with Python, without pip, and with the Hatch binary sounds about right.
I'm guessing that the ultimate issue here is that hatch-pip-compile is actually trying to install uv into the virtual environment with pip, so that uv can be used as the installer/resolver. I implemented the uv functionality in hatch-pip-compile before it actually arrived to hatch - now that uv can ship with hatch I should be able to use the uv from hatch's own environment to handle all of this and have no need for pip in your case.
pip-tools (being used as a Python library, not the CLI), is used to parse a requirements file into a list of packaging.requirements.Requirement so that this can be passed to hatch's dependencies_in_sync check. This is what lets hatch know whether the right set of direct and indirect dependencies installed.
Feel free to read through this discussion here if you're curious why private modules were used: https://github.com/juftin/hatch-pip-compile/pull/51#discussion_r1434240582.
Ah! This is an issue with hatch and not hatch-pip-compile (cc/ @ofek). The issue is that hatch is trying to use pip to install hatch-pip-compile (tool.hatch.env.requires).
See this Dockerfile where python and hatch are available but not pip:
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "foo"
version = "0.0.1"
[tool.hatch.env]
requires = [
"hatch-pip-compile"
]
[tool.hatch.envs.default]
installer = "uv"
Dockerfile
FROM fedora:latest
RUN dnf install -y python3 hatch
WORKDIR /work
RUN touch /work/pyproject.toml
RUN cat <<EOF > /work/pyproject.toml
[build-system]
requires = ["hatchling"]
build-backend = "hatchling.build"
[project]
name = "foo"
version = "0.0.1"
[tool.hatch.env]
requires = [
"hatch-pip-compile"
]
[tool.hatch.envs.default]
installer = "uv"
EOF
RUN mkdir /work/foo
RUN touch /work/foo/__init__.py
CMD ["hatch", "run", "ls"]
docker build . --tag hatch-pip-compile-test
docker run --rm -it hatch-pip-compile-test
/usr/bin/python3: No module named pip
This would be an issue with Hatch indeed but I'm not sure the behavior people desire exactly. The issue is that plugins currently must live in the same installation as Hatch so if there is no pip the assumption fails. For example, if you change that installation line to also add python3-pip it passes. Configuring UV inside environments shouldn't matter at that external level because they are separate concepts.
So I guess there are three options:
- Add explicit UV configuration to
tool.hatch.env - Try UV if it exists on PATH followed by the current
sys.executable -m pip - Manage plugins in a separate internal environment and do some weird PYTHONPATH stuff
What do you think? The final one would be interesting because the plugins would persist across Hatch updates but would be prone to errors because if any dependencies are not pure Python then upgrading the Python version with which Hatch is running would cause issues so special care must be taken.
Thanks for the thoughtful response as always!
For the above example (without python3-pip), where the user only wants uv and doesn't want pip, I think it's reasonable for the following to work (option 1):
[tool.hatch.env]
requires = [
"hatch-pip-compile"
]
installer = "uv"
As an alternative, since uv is a dependency of hatch we should know for certain it's available within the same environment as hatch. Could we use that (instead of whatever pip / uv might exist on the PATH outside the hatch env) to bootstrap our hatch.env.requires dependencies?
Those are some interesting insights. I appreciate both of you taking the time to think about this.
@ofek I am not familiar enough with the Hatch source yet, but if you could point me to a function/module that might be a good starting point for this particular issue, I am willing to give it a shot. Since I brought up this edge case and you surely have enough to do already. After a cursory search I could not find the place in code, where the hatch.env.requires setting is processed and the plugins are actually installed.
Stuck with the same error. As a workaround, someone may use the following command to add pip for hatch to be used:
uvx --with pip hatch run ls
Hello everyone,
I created a PR in the hatch repo pypa/hatch#1882 that should resolve the described issue. Tests are passed on my end and I checked on one of my simple projects with hatch-pip-compile plugin enabled that the issue is resolved.
I would appreciate though if someone can also test that hatch version with fix on theirs project and provide a feedback as well. 😅