[red-knot] PEP 561 compliant module resolver

[carljm]

trafficstars

(Or intentional choices to diverge from PEP 561 compliance.)

Subtasks:

• [x] Vendor typeshed's stubs: https://github.com/astral-sh/ruff/pull/11340
• [x] Add automation for keeping typeshed's stubs up to date: https://github.com/astral-sh/ruff/pull/11427
• [x] Implement --custom-typeshed-dir internally: https://github.com/astral-sh/ruff/pull/11767
• [x] Implement the typing spec's module resolution order:
  • https://github.com/astral-sh/ruff/pull/11767
  • Proposal for tweaking the spec posted at https://discuss.python.org/t/pep-561-s-module-resolution-order-seems-incorrect/55048/1
• [x] Make vendored typeshed stubs available from the built Ruff binary: https://github.com/astral-sh/ruff/pull/11779
• [ ] Use vendored typeshed stubs at runtime to resolve symbols to stdlib definitions if --custom-typeshed-dir is not passed
  • [x] Add a VendoredFileSystem implementation: https://github.com/astral-sh/ruff/pull/11863
  • [ ] Use the VendoredFileSystem implementation to resolve stdlib stubs: #12224
• [ ] Add support for stubs packages suffixed with -stubs installed into site-packages (must take priority over other packages installed into site-packages)
• [x] Add support for typeshed's VERSIONS file: we shouldn't fallback to a typeshed module if the VERSIONS file specifies it does not exist with the --target-version the user passes.
  • [x] Add a parser for VERSIONS: https://github.com/astral-sh/ruff/pull/11836
  • [x] Use the VERSIONS file when resolving stdlib modules: https://github.com/astral-sh/ruff/pull/12141
• [ ] Audit mypy's and pyright's module resolvers to see if there's anything that's not in the spec that still needs to be implemented

[MichaReiser]

One thing that we need to implement as part of this is a vendored filesystem. It's not a functionality but a task to complete on our way to get there.

[AlexWaygood]

One thing that we need to implement as part of this is a vendored filesystem. It's not a functionality but a task to complete on our way to get there.

Right. I was considering that part of

Use vendored typeshed stubs at runtime to resolve symbols to stdlib definitions if --custom-typeshed-dir is not passed

since, as you say, it's necessary to implement a vendored filesystem in order to be able to do that

[MichaReiser]

@AlexWaygood is there more work that needs doing or can we consider this done?

[AlexWaygood]

We can't call ourself PEP-561-compliant until we do this, which is still outstanding, and important. It's not huge, but also not trivial:

* [ ]  Add support for stubs packages suffixed with `-stubs` installed into `site-packages

But we decided to defer work on it for now because the type-inference work is more important.

We should also add support for py.typed files that explicitly mark type annotations as partial, which is specified by the PEP but not listed in the tasks above; I'll add that now.

[carljm]

Finer-grained issues are easier to manage for prioritizing and tracking work to be done, so I've created https://github.com/astral-sh/ruff/issues/16612 and https://github.com/astral-sh/ty/issues/184 for the last two items here.

I don't think "audit mypy and pyright resolvers" is likely worth the trouble, it will be more efficient to discover issues from encountering them in the wild. If someone disagrees, feel free to create an issue for it! (Or just do it and create issues for anything observed.)