python-poetry
Question about export of dependency with extra, e.g. httpx[http2], in Python 1.8.5 vs this plugin
I attempted to create two identical projects, one using Poetry 1.8.5 and the other using Poetry 2.1.1 with this plugin.
The folder trees of both are analogous:
│ poetry.lock
│ pyproject.toml
│ README.md
│ reqs.txt
│
└───src
└───one_eight_five
__init__.py
with this pyproject.toml
[tool.poetry]
name = "one-eight-five"
version = "0.1.0"
description = ""
authors = ["Your Name <[email protected]>"]
readme = "README.md"
packages = [{include = "one_eight_five", from = "src"}]
[tool.poetry.dependencies]
python = "^3.13"
httpx = {extras = ["http2"], version = "*"}
[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"
vs
│ poetry.lock
│ pyproject.toml
│ README.md
│ reqs.txt
│
└───src
└───two_one_one
__init__.py
with this pyproject.toml
[project]
name = "two-one-one"
version = "0.1.0"
description = "test"
authors = [
{name = "Your Name",email = "[email protected]"}
]
readme = "README.md"
requires-python = ">=3.13,<4.0.0"
dependencies = [
"httpx[http2]"
]
[tool.poetry]
requires-poetry = ">=2.1"
packages = [{include = "two_one_one", from = "src"}]
[tool.poetry.requires-plugins]
poetry-plugin-export = ">=1.8"
[build-system]
requires = ["poetry-core>=2.0.0,<3.0.0"]
build-backend = "poetry.core.masonry.api"
I did a poetry export --without-hashes --without-urls of both projects and have attached the output below. The question is that the version 1.8.5 export has this line:
httpx[http2]==0.28.1 ; python_version >= "3.13" and python_version < "4.0"
whereas the version 2.1.1 export has this line:
httpx==0.28.1 ; python_version >= "3.13" and python_full_version < "4.0.0"
Note that the [http2] is missing. Since both requirements files have a line for h2, which is all that [http2] ends up meaning for the httpx package, I guess I haven't lost anything due to that missing [http2] if I go use the requirements file in a Dockerfile build process. But the difference was confusing to me, and I was hoping to hear from someone who knows how this is supposed to work. Ought the two requirements files show the same thing?
When I poetry build both projects, the .whl files' METADATA each contain this line:
Requires-Dist: httpx[http2]
I would say this is expected and does not matter because all dependencies are resolved. (You should always install an exported requirements.txt with --no-deps anyway.) We even have a test case for that behavior:
https://github.com/python-poetry/poetry-plugin-export/blob/875cda2aa22fe2e4b36bd50b4bf4ae62b20d21af/tests/test_exporter.py#L2409-L2425
With old lock files, the exporter has to do some kind of resolution and knows the requested extras from the pyproject.toml.
With new lock files, the exporter just iterates over the locked packages - and the lock file does not contain information which extras have been requested.
I think we could change it back if there is a good reason but it will make the export code more complicated.
I really appreciate the quick response. Thank you! We're about to convert a bunch of repos over to poetry 2.1.1 and I wrote a script to try to validate the new pyproject.toml and this was the only lingering concern, which you have addressed.
Unfortunately, I don't understand your parenthetical "You should always install an exported requirements.txt with --no-deps anyway." The pip doc explains that as "Don’t install package dependencies." But if I'm using poetry export to capture a requirements.txt to feed to a Dockerfile build process (for example, I'm on a Windows box but I need my packages for a Linux AWS Lambda Layer deployment), don't I need to install the dependencies in the Linux container to make sure I have a Linux-happy set of packages?
I am obviously confused about what --no-deps means. I just read this article https://www.b-list.org/weblog/2023/dec/07/pip-install-safely/, and, assuming that it is correct, I am less confused. Quoting it
--no-deps tells pip not to try to resolve and find and install dependencies for the packages you’ve specified. It shouldn’t need to, since your requirements file should contain the full dependency tree already resolved and pinned to exact versions, so all pip should need to do is fetch and install that set of packages. And though this does require you to stick to the process and ensure you provide a fully-resolved dependency set in your requirements file, it also shuts down one way you might encounter surprises during installs (as can sometimes happen, and not only due to malice — if newer versions of some transitive dependencies have been published, for example, and pip tries to do a fresh resolution, it might see those newer versions, prefer them, and then fail because they weren’t the exact pinned-and-hashed versions given in the requirements file). It’s also a little bit faster since dependency resolution is an expensive step.
If that article is correct, no need to respond to my confusion! Again, I appreciate your answer about the missing [https]` as, my parapharse to test my comprehension, a cosmetic difference between this plugin and the old poetry functionality.
The exported requirements.txt includes all direct and transitive dependencies. Further, it represents a universal solution for all possible environments so even if you export on your Windows box and there is a package you only need on Linux, this package will be included with a marker like sys_platform == "linux" or similar.
The quoted paragraph is correct and also applies to requirements.txt files exported with poetry.
This does matter, if the library httpx[http2] itself is an optional dependency. Creating a group with optional dependencies and then using --all-groups is a workaround i can work with
@twsl , for the non-poetry experts can you share a snippet that make optional dependency working?
