Add support for the `project` section (PEP 621)

[radoering]

Pull Request Check List

Resolves: #3332 Resolves: python-poetry/roadmap#3 Requires: python-poetry/poetry-core#708

should not be squashed

• [x] Added tests for changed code.
• [x] Updated documentation for changed code.

Current state:

• Full compatibility with python-poetry/poetry-core#708
• All the commands should work, especially check, lock, install, build, add, remove, init, new
• Updated documentation:
  • See preview
  • Explanation of project fields and use cases for dynamic (version, readme, classifiers) in the section "The pyproject.toml file".
  • Explanation of the difference between project.dependencies and tool.poetry.dependencies in the section "Dependency specification".
  • Updated examples to use project fields instead of tool.poetry fields.

You can install this branch with pipx as follows:

pipx install --suffix _pep621 git+https://github.com/radoering/poetry.git@pep621-support

This will install Poetry with the suffix "_pep621" so that you have to call poetry_pep621 instead of poetry. (The suffix is arbitrary, you can choose a different one.)

There is no automatic conversion of existing pyproject.toml files (yet?) but poetry check will give you information which fields should be transferred from tool.poetry to project.

[github-actions[bot]]

Deploy preview for website ready!

✅ Preview https://website-870gzwxbb-python-poetry.vercel.app

Built with commit 28bbbf3ee21d5e9ff82f001a85b9aa803a6a5fa3. This pull request is being automatically deployed with vercel-action

[johnthagen]

After this is released, Poetry can be added to the PyPA packaging tutorial. 🚀

• https://packaging.python.org/en/latest/tutorials/packaging-projects/

[Evan-Zhao]

First of all, thank you for the great work!

I'm a bit confused by the entire extra business (which is pretty complicated itself), and especially how poetry add interacts with extras. Disclaimer: I may have wrong understanding of extras or optionals.

For this example pyproject.toml, say now I would like to add an extra feature called "dev" with dependencies black and Cython. This way pip install tvm[dev] will also install black and Cython, which a tvm developer will need (but not general users). In other words I want to do poetry add ??? black Cython, and Poetry should add such a line:

[project.optional-dependencies]
# ...
dev = ["cython (>=3.0.10,<4.0.0)", "black >= 24"]   # added

So I go to poetry add --help and find two relevant options, but it turns out neither does what I want here. One is --optional, which specifies the dependencies are optional using a tool.poetry.dependencies specification. Makes perfect sense. The other one, --extras <extra>, is harder to follow.

Suppose I already work with tvm[highlight] (that is, with the extra highlight on) in my own environment. If I do poetry add <dep> without -E highlight, Poetry starts to remove highlight-related deps from my virtual env, which tells me -E specifies the set of extras to consider when computing the new deps. It wasn't clear to me from the help, but this behavior is logical. But now I do poetry add -E highlight Cython, and Poetry adds this:

dependencies = [
  # ...
  "cython[highlight] (>=3.0.10,<4.0.0)"]  # added

where highlight becomes cython's extra. Is that expected? I thought if I want to add a dep with an extra I'd do poetry add "dep[extra]" instead.

It looks to me we need a command that "given existing extras e_1, e_2, ... e_n, add deps dep_1, dep_2, ..., dep_n into an optional extra e_new (if not given then add to global dependencies)", so something like poetry add [--current-extras e]+ [--target-extra e]? [dep]+ (not actually proposing this syntax).

• In developing my project I keep multiple current-extras on, and it's kind of clumsy to always enter all of them for each add command -- is it possible at all to auto-deduce that from the lock file?
• Also I can't speak for how all of this interact with Poetry optionals and dep groups, since I'm not using these yet, but I'd love to help discuss should complexity arises there.

[radoering]

If I do poetry add without -E highlight, Poetry starts to remove highlight-related deps from my virtual env, which tells me -E specifies the set of extras to consider when computing the new deps.

That's unfortunate but not easy to change as we do not keep track of which extras (of your project) were installed. Furhter, it is not specific to this PR and thereby out of scope here. I suppose there is already an issue about this but I can't find it right now. Feel free to create a new issue. (In case it's a duplicate someone will tell you.)

If I do poetry add without -E highlight, Poetry starts to remove highlight-related deps from my virtual env, which tells me -E specifies the set of extras to consider when computing the new deps.

This conclusion is not correct. poetry add does not consider extras of your project for installing. You can probably work around this by using poetry add --lock ... to only lock (and not install in your environment) and in the end run poetry install -E highlight to install everything including your project's extra.

where highlight becomes cython's extra. Is that expected?

That's expected. See poetry add --help or the docs: "Extras to activate for the dependency."

I thought if I want to add a dep with an extra I'd do poetry add "dep[extra]" instead.

Both are equivalent.

It looks to me we need a command

--optional just adds an optional dependency but does not add it to an extra. Since that does not make sense anymore in the project section, I changed --optional to require an extra name to which the dependency is added, so now you can do: poetry add --optional highlight Cython to add Cython to the extra highlight.

In developing my project I keep multiple current-extras on, and it's kind of clumsy to always enter all of them for each add command -- is it possible at all to auto-deduce that from the lock file?

In the lockfile, only extras of dependencies are fix. All extras of your project are locked and you can/must decide which to install when running poetry install.

Also I can't speak for how all of this interact with Poetry optionals and dep groups, since I'm not using these yet

which a tvm developer will need (but not general users).

Maybe, you should consider using dependency groups instead of extras. However, dependency groups are only available during development when using Poetry and not visible in the sdist/wheel or to other tools.

All in all, there is some room for improvement considering extras but everything that goes beyond my recent change regarding --optional should probably be discussed in a separate issue.

[edgarrmondragon]

This seems to be working great on our end: https://github.com/meltano/sdk/pull/2407

Even mtkennerly/poetry-dynamic-versioning works smoothly in conjunction with dynamic = ["version"].

[hardchor]

Can confirm this works with https://github.com/wandb/openui/blob/main/backend/pyproject.toml :clap:

[z3z1ma]

Can confirm this works on a fairly complex internal project with a lot of deps. Very nice. We are using package-mode = false and leveraging just lockfile based dep management.

[deronnax]

Can also confirm this works on a fairly complex internal project with a lot of deps (40 direct deps, 332 total deps)

[edgarrmondragon]

I actually encountered one small issue when testing 8f98ee2b9a0d6e6aee05ca000f29fa5cd113fcaa:

• Setting name only in [project] causes a KeyError when running poetry check.

[!NOTE] UPDATE: on a closer look at the traceback, this actually seems to be caused by poetry-dynamic-versioning

Full traceback

  Stack trace:

  8  ~/.local/pipx/venvs/poetry-pep621/lib/python3.12/site-packages/cleo/application.py:327 in run
      325│ 
      326│             try:
    → 327│                 exit_code = self._run(io)
      328│             except BrokenPipeError:
      329│                 # If we are piped to another process, it may close early and send a

  7  ~/.local/pipx/venvs/poetry-pep621/lib/python3.12/site-packages/poetry/console/application.py:180 in _run
      178│         self._load_plugins(io)
      179│ 
    → 180│         exit_code: int = super()._run(io)
      181│         return exit_code
      182│ 

  6  ~/.local/pipx/venvs/poetry-pep621/lib/python3.12/site-packages/cleo/application.py:431 in _run
      429│             io.input.interactive(interactive)
      430│ 
    → 431│         exit_code = self._run_command(command, io)
      432│         self._running_command = None
      433│ 

  5  ~/.local/pipx/venvs/poetry-pep621/lib/python3.12/site-packages/cleo/application.py:473 in _run_command
      471│ 
      472│         if error is not None:
    → 473│             raise error
      474│ 
      475│         return terminate_event.exit_code

  4  ~/.local/pipx/venvs/poetry-pep621/lib/python3.12/site-packages/cleo/application.py:454 in _run_command
      452│ 
      453│         try:
    → 454│             self._event_dispatcher.dispatch(command_event, COMMAND)
      455│ 
      456│             if command_event.command_should_run():

  3  ~/.local/pipx/venvs/poetry-pep621/lib/python3.12/site-packages/cleo/events/event_dispatcher.py:26 in dispatch
       24│ 
       25│         if listeners:
    →  26│             self._do_dispatch(listeners, event_name, event)
       27│ 
       28│         return event

  2  ~/.local/pipx/venvs/poetry-pep621/lib/python3.12/site-packages/cleo/events/event_dispatcher.py:85 in _do_dispatch
       83│                 break
       84│ 
    →  85│             listener(event, event_name, self)
       86│ 
       87│     def _sort_listeners(self, event_name: str) -> None:

  1  ~/.local/pipx/venvs/poetry-pep621/lib/python3.12/site-packages/poetry_dynamic_versioning/plugin.py:174 in _apply_version
      172│         io = _should_apply_with_io(event.command.name)
      173│ 
    → 174│         _apply_version_via_plugin(self._application.poetry, io=io)
      175│         _patch_dependency_versions(io)
      176│ 

  KeyError

  'name'

  at ~/.local/pipx/venvs/poetry-pep621/lib/python3.12/site-packages/poetry_dynamic_versioning/plugin.py:85 in _apply_version_via_plugin
       81│     io: bool = True
       82│     # fmt: on
       83│ ) -> None:
       84│     name = _get_and_apply_version(
    →  85│         name=poetry.local_config["name"],
       86│         original=poetry.local_config["version"],
       87│         pyproject=poetry.pyproject.data,
       88│         pyproject_path=_get_pyproject_path_from_poetry(poetry.pyproject),
       89│         retain=retain,

• Setting name both in [project] and [tool.poetry] raises a warning.

Warning: [project.name] and [tool.poetry.name] are both set. The latter will be ignored.

[hexmode]

When I use this patch on this project with this change:

@@ -1,4 +1,4 @@
-[tool.poetry]
+[project]
 name = "poetry"
 version = "1.9.0.dev0"
 description = "Python dependency management and packaging made easy."

I get

$ poetry run ls

The Poetry configuration is invalid:
  - project.license must be valid exactly by one definition (0 matches found)

[radoering]

@hexmode That's because tool.poetry.license is a string but project.license must be a table according to the spec. You cannot just replace [tool.poetry] with [project] since not all tool.poetry fields are valid project fields. See the docs preview at https://github.com/python-poetry/poetry/pull/9135#issuecomment-1987233477 (section "The pyproject.toml file").

[huynguyengl99]

Hi, when will we have this feature available in Poetry? I assume this update will be part of Poetry 2.0, but does the team have a rough estimate for the release? I just want to know because I'm building a small package and currently using poetry @ git+https://github.com/radoering/poetry.git@pep621-support for easier migration to Poetry 2.0 in the future, so I'm curious about its release. Thank you if you can share the timeline in advance.

[radoering]

Hi, when will we have this feature available in Poetry? I assume this update will be part of Poetry 2.0, but does the team have a rough estimate for the release?

As always, there is no schedule but yes, this will be part of 2.0 or in other words: without this the next version will probably not be 2.0. To make it more transparent I created #9448 to track the progress for 2.0. In my assessment, the main features have been implemented so far and "only" need a review. (However, that might take a while because these are big changes and the available time of other members of the core team seems to be rather limited at the moment.)