Revert "Revert "Adapt scheduler to work with dynamic derivations

[Ericson2314]

Motivation

This fixes dynamic derivations, reverting #9081. #9052 However will be reintroduced unless this is modified somehow.

Context

This reverts commit 8440afbed756254784d9fea3eaab06649dffd390.

Priorities

Add :+1: to pull requests you find important.

[dimbleby]

this is a variation on "recording package.optional in poetry.lock is a bad idea". https://github.com/python-poetry/poetry/issues/3188 is another example.

in this case it is get_extra_package_names() that does not recognise that importlib-metadata is required by the sphinx extra: because it sees that extra and assumes that the first sphinx it finds will be just fine for the rest of its analysis. But the first sphinx it finds is the newer one that does not need importlib-metadata, and after that it never considers the other.

the Right Way to fix this IMO is not to put optional in the lockfile at all: this is a misunderstanding of what the lockfile really is. Rather the installer-solver ought to start out knowing what extras it is solving for, and just return the right answer.

I expect that sorting that out will be quite the tangle for someone.

[radoering]

Might be fixed in the main branch.

1.8.3:

$ poetry install --dry-run --all-extras | grep importlib-metadata
  - Installing importlib-metadata (8.2.0): Skipped for the following reason: Not required

main:

$ poetry install --dry-run --all-extras | grep importlib-metadata
  - Installing importlib-metadata (8.2.0)

[dimbleby]

Pretty sure it isn't, maybe the luck of which sphinx happens to show up first during get_extra_package_names() is unstable

[dimbleby]

Apparently I knew about this at https://github.com/python-poetry/poetry/pull/6372#issue-1360542275

[Secrus]

I am going to consider this a solved issue.

[dimbleby]

This is not a solved issue