Support selecting Python version via `tool.poetry.dependencies.python`

[edmorley]

Initial support for Poetry was added in #7, and included support for bootstrapping Poetry and then using it to install app dependencies.

This issue is for adding support for controlling the Python version via the tool.poetry.dependencies.python TOML field in pyproject.toml when using Poetry. This will be in addition to the existing ability to use runtime.txt, and the planned ability to use .python-version (see #6).

An example Poetry config created using poetry init with when using Poetry 1.8.3 and Python 3.11 contains:

[tool.poetry.dependencies]
python = "^3.11"

However, the next release of Poetry is due to change the default constraint marker from ^ to >= (see https://github.com/python-poetry/poetry/pull/9558), which will give:

[tool.poetry.dependencies]
python = ">=3.11"

It's worth noting that:

• Poetry's dependency syntax doesn't use PEP-440 style versioning, but instead something closer to the Node.js style semver syntax. This means it differs from the syntax used in requirements files and in other non-Poetry parts of pyproject.toml (including the project.requires-python field, which is what uv uses), which makes things more complicated both for the buildpack and for user UX (for example both syntaxes have a specifier that includes the tilde character, but they do drastically different things for some edge cases). See: https://python-poetry.org/docs/dependency-specification/
• The Python version used in the output of poetry init uses the version of the local Python version to set the minimum Python version.
• If the python field isn't specified at all, Poetry defaults to a wide range (currently >=2.7,<2.8 || >=3.4), which then breaks installing any packages that have a smaller compatibility range than that (ie: any package that only supports Python 3) - since Poetry treats its python field as a "this project must be compatible with all of these Python versions" value. As such, omitting python really isn't viable for users - so it's going to be set most of the time.
• Both the "^3.11" and ">=3.11" forms allow higher major versions of Python such as 3.12 or 3.13, which can include breaking changes. Whilst supporting a range of versions makes sense for a library, for applications these unbounded ranges can cause issues (as we've seen with the Node.js buildpacks over the years), and so ideally we wouldn't want apps to use these forms.

All of the above means that adding support for tool.poetry.dependencies.python is going to mean making compromises in one area or another sadly, since we either have to:

1. Explicitly support "unsafe" version ranges such as those above. (Pros: Compatibility with the default output of poetry init. Cons: Breaking changes when new major Python versions are released + encourages environment drift between CNB and local development environments - and even from one developer's machine to another.)
2. Ignore the tool.poetry.dependencies.python field (either completely, or perhaps only if it uses an unsafe range) and instead install the buildpack's curated default Python version. (Pros/cons: Pretty much the same as (1), apart from the curated Python version perhaps being marginally more compatible with packages in the wild, since we wait a couple of months before making new Python versions the default.)
3. Partially support "unsafe" ranges, by using the lower bound of the range as the Python version to be installed, rather than the upper bound. (Pros: Compatibility with the default output of poetry init and avoids breakage when new Python versions are released. Cons: Doesn't prevent environment drift - developers can still be using a different Python version locally - particularly as new versions are released, unless they remember to bump the project's minimum Python version.)
4. Error on "unsafe" version ranges telling users to change to a stricter range instead (eg 3.11.*). (Pros: Prevents breaking changes when new versions of Python released + prevents environment drift. Cons: First build on any app using Poetry will likely fail, unless they are using a template that already has a stricter range set.)

At the moment I'm leaning towards (4), since:

• Our priority should be encouraging users towards safe patterns, rather than those that can cause hard to debug issues later (we already get support tickets where users using pip+runtime.txt were using different version of Python locally vs on Heroku and blame Heroku saying "it works on my machine")
• Users using a Python package/project manager that uses a lockfile have choosing to use a more advanced tool that offers determinism over short-term convenience. Enforcing that a safe range is used (that forces use of a specific major Python version) would be in line with that.
• We can always make .python-version take priority over tool.poetry.dependencies.python, and so the error message can say to either adjust the tool.poetry.dependencies.python range or create a .python-version file as an alternative (if they want to keep the wide range in the Poetry config).
• Poetry's init command supports a --python option, so we could always encourage users to use poetry init --python '3.12.*' in our docs to save them having to fix the default version afterwards.
• We could always advocate for Poetry supporting an "--type app" (or similar) for poetry init which picks defaults more appropriate for an app vs a library (in addition to it setting a Python version like 3.11.* instead of an unbounded range, it could also set package-mode = false which would avoid all of the other boilerplate)

We'll also want to factor in the pyproject.toml project.requires-python field since that is what uv uses and so we may need to support that too in the future.

See also:

• https://python-poetry.org/docs/basic-usage/#setting-a-python-version
• https://python-poetry.org/docs/dependency-specification/
• https://github.com/python-poetry/poetry/issues/3332
• https://github.com/heroku/buildpacks-python/issues/9
• Not applicable to Poetry's syntax, but useful for context:
  • https://packaging.python.org/en/latest/specifications/version-specifiers/#id5 (the spec from PEP-440 and others)
  • https://packaging.python.org/en/latest/specifications/pyproject-toml/#requires-python (the spec from PEP-621 and others)
  • https://github.com/heroku/buildpacks-python/issues/6
  • https://github.com/astral-sh/uv/issues/7429

GUS-W-9608268.

[edmorley]

We could always advocate for Poetry supporting an "--type app" (or similar) for poetry init which picks defaults more appropriate for an app vs a library (in addition to it setting a Python version like 3.11.* instead of an unbounded range, it could also set package-mode = false which would avoid all of the other boilerplate)

I've filed a feature request for this upstream:

• https://github.com/python-poetry/poetry/issues/9668

[edmorley]

Presuming we pick option (4), then the tool.poetry.dependencies.python specifiers we'll support will be:

• 3.12.*
• 3.12.5
• ==3.12.*
• ==3.12.5

We would then reject anything else with an error message that says to use one of the above, or to add a .python-version file instead (the .python-version file will accept values like 3.12 or 3.12.5).

Examples of specifiers we would then reject:

• *
• 3.*
• >=3.12
• ^3.12
• ~3.12.5
  • whilst this is a "safe" range, it's (a) very similar to the PEP-440 ~= specifier, which has quite different semantics for certain cases (for example whilst ~3.12 and ~3.12.0 are equivalent, ~=3.12 and ~=3.12.0 are not), (b) I suspect there may be Python users who aren't used to seeing the Node.js semver style tilde usage and think this specifier is actually equivalent to 3.12.5, (c) it would needing to do actual version resolution rather than a simple mapping to "latest 3.12.x" (which adds to complexity for minimal benefit).
• >=3.12,<3.13
  • supporting complex specifiers (that have multiple clauses) would mean needing to do actual version resolution (which adds to complexity for minimal benefit)

[edmorley]

To add yet more things to think about - Poetry has just merged support for PEP-621 and the [project] section of pyproject.toml, in: https://github.com/python-poetry/poetry/pull/9135

This means Poetry 2.0 (expected later this year, see https://github.com/python-poetry/poetry/issues/3332#issuecomment-2351546603 and https://github.com/python-poetry/poetry/issues/9448) will support requires-python too.

Given that:

• Poetry supporting PEP-621 means it will presumably be slowly moving away from its proprietary tool.poetry.* table
• We could only ever support a limited subset of the tool.poetry.dependencies.python syntax (syntax which isn't the default, so the first run experience will always be an error)
• uv is leaning more into using .python-version for picking the single Python version to install (rather than requires-python)

... it's making me wonder whether we should instead double down on .python-version and not support tool.poetry.dependencies.python at all?

[edmorley]

Given all of the above, I'm wontfixing this for now in favour of having a single way to specify the Python version - using a .python-version file.