Separate and clearly document direct/transitive vendored dependencies with the help of `pip-tools`

[abravalheri]

The work in this PR was extracted from #4330.

Motivation

Right now it is a bit tricky to deal with the vendored packages:

• The developer either have to manually pin down all the direct and transitive dependencies in vendored.txt or
• There will be dependencies installed in the _vendor folder that are not listed in the vendored.txt file.
  • This makes it is hard to guess why they are there and to track the dependency chain...
  • It also make it possible for pkg_resources and setuptools to have different versions of the same vendored dependency. This is probably not nice for OS-level repackaging.

Summary of changes

The most relevant commit of this PR is b5e91c836.

It modifies tool/vendored so that:

1. Transitive dependencies are automatically documented (so we don't loose track of what needs what).
   • This is done via pip-compile and vendored.in (direct dependencies) ==> vendored.txt. See 1a31ed107 and 20df694e3.
2. Instead of a case-by-case import rewrite, all vendored imports are rewritten at once
   • Case-by-case workarounds were kept (e.g. intentionally delayed imports)
3. Sync versions of vendored packages in pkg_resources and setuptools so that it is easier for "de-vendoring" (this is done via --constraint in pip).

I also added a base_python to the tox environment intended to match the lowest version of Python supported by setuptools, so that dependency markers are solved for a more general case.

The remaining of the changes are collateral.

No attempt to remove extern was made.

If reviewing this PR I recommend checking commit-by-commit to avoid the big blob of automatic changes related to running tox -e vendor.

Closes

Pull Request Checklist

• [ ] Changes have tests
• [ ] News fragment added in newsfragments/. (See documentation for details)

[davidhewitt]

No attempt to remove extern was made.

I was just reviewing https://github.com/PyO3/setuptools-rust/pull/437 and trying to assess the suggestion to switch from using tomli as a real dependency to pull it from setuptools.extern (on Python versions predating tomllib).

What's your take on this situation? It seems a reasonable motivation for setuptools-rust to avoid having to pull in the additional dependency, though based on the PR here and previous discussions it's based upon, it seems that setuptools was not intending for setuptools.extern to be public...

[eli-schwartz]

Note that doing so is a little bit naughty, but it's something I've done before, e.g. https://pagure.io/389-ds-base/pull-request/51041

And seen done elsewhere: https://github.com/Python-Markdown/markdown/issues/825#issuecomment-492906966

My belief is that it is safe to do as long as you try both, under the assumption that it's a useful enough private design that the extern import will stay around for as long as the copy is vendored, and once it isn't vendored the other part of the try/except will succeed anyway.

I don't generally recommend using it anyway but for this specific case it felt apropos given the use case was a setuptools plugin which has tight runtime coupling to setuptools -- and setuptools has no public API for loading pyproject.toml into a dict, something that most plugins need to do, which means it's either this or requiring their own "tomli" dependency.

[abravalheri]

I will close this since setuptools changed the way it process its vendored dependencies.