Use declarative metadata with `pyproject.toml`

[AA-Turner]

Requires #10355, #10352

This moves Sphinx's package metadata to the standards-compliant format, preserving setuptools-specific items with the tool.setuptools backend.

Relates

• https://blog.ganssle.io/articles/2021/10/setup-py-deprecated.html

[AA-Turner]

It seems that tox requires fully building and installing the package, and setuptools dropped support for Python 3.6 in December 2021 (https://setuptools.pypa.io/en/latest/history.html#v59-7-0), before the support for pyproject.toml was added.

I'm going to mark this as draft for now, I may look at e.g. flit later (that Sphinx uses egg_info configuration slightly complicates things, though).

A

[AA-Turner]

It was easier than expected to move to flit, so I'm un-marking as draft. CI is passing.

A

[nanonyme]

To be fair, Python 3.6 is since EOL this January.

[tk0miya]

As I commented at https://github.com/sphinx-doc/sphinx/issues/9820#issuecomment-1140500002, my large concern is supporting users. I suppose most of them would not know python at all. So I hesitate to depend on the latest libraries.

I can accept to migrate package metadata that is not related to install itself (ex. classifiers). But I'm not sure it's worthy for us to migrate it partially.

[nanonyme]

Users using pip will not notice any difference with this.

[AA-Turner]

As I commented at https://github.com/sphinx-doc/sphinx/issues/9820#issuecomment-1140500002, my large concern is supporting users. I suppose most of them would not know python at all. So I hesitate to depend on the latest libraries.

I entirely agree we should be supporting users. I see three groups:

1. "python user" -- uses pip install sphinx
2. "distro user" -- uses apt install sphinx (or yum, dnf, etc.)
3. "sources user" -- uses git clone https://github.com/sphinx-doc/sphinx.git && cd sphinx && python setup.py build or similar

For group 1, pip manages everything automatically. It fetches pacakges matching the constraints that we specify, and ensures a consistent installation.

For group 2, the redistributor (debian, RHEL, ubuntu, etc) ensures that the system has a consistent installation, and installs everything themselves.

For group 3, there are no guarantees at all -- it is up to the user to ensure every dependency is installed and there are no conflicts.

I would argue that the users of Sphinx are mostly in group 1, then group 2, then 3. There are five million downloads of Sphinx per month from PyPI. We can't know numbers for redistributors, or proxies of PyPI, so this is an inexact science.

Group 2 users are in a precarious position when they attempt to use extensions like autodoc -- as Sphinx needs to import the module being documented, all the module's dependencies must be present. This might create conflicts with other Python packages in the distribution's global installation, which is why environments of some kind (venv, virtualenv, conda env) are so strongly reccomended.

I would like to solve these challenges -- having a single distributable self-contained binary that is "sphinx" would be amazing. It would free us from constraints on system Python versions etc, whilst allowing integration with a particular python version for those that install via pip. However we don't have this (sadly).

My conclusions are the following:

• pip users will be fine with any sort of version constraints, as pip is a proper package manager.
• users of distribution (ubuntu etc) supplied Sphinx binaries will be fine, as distributions manage all their versions independently of us, often pinning more strictly etc to ensure that the system as a whole has a consistent Python installation.
• Hence, we should not worry about redistributors.
• Users who install from source are assumed to be experts, and we do not provide any extra support here.

---

The issue in this PR -- changing our build process -- is unrelated to any of these issues. For users of pip, pre-build wheels are downloaded from PyPI. For distribution users, only the packaging team need to care about our build process, and all distributions have PEP 517 / 518 tool chains now. From this PR, nothing will change for end users.

---

I'm happy to propose a PR to Sphinx's policies @tk0miya to outline this and provide a place for more discussion, but I would strongly argue that we can make these important improvements and changes in our build system and dependencies without negatively affecting end users, which is the most important thing.

A

(Sorry for the long message, I didn't intend to write this much!)

[nanonyme]

Note in case 3 you can do pip install . and it becomes case 1 again. The real case 3 is offline install, that is, users who manually download all deps.

[danieleades]

mypy config should also be moved into pyproject.toml file

[danieleades]

as an aside, is there any interest in moving to a full-blown package manager, such as Poetry?

[AA-Turner]

I would like to move forwards with this--@tk0miya does my earlier comment make sense?

The one line summary is that I propose we only support users installing from wheels on PyPI -- anything else is non-standard, and as we are volunteers here I think reasonable to put those limits on it.

A

[AA-Turner]

I've gone ahead with this for 5.2--though if we get reports from users that the new configuration seriously breaks things I will look at reverting the change.

A