Modernize build system

[pganssle]

Since I really appreciate how clean and well-maintained this repository is, I figured I'd do a quick sweep through to help modernize the build system. The changes are explained in detail in each of the commit messages, but the overview is:

1. Direct invocation of setup.py install is soft-deprecated (it's a bad idea to do it, but actually raising warnings in only the right situation is currently a bit tricky), so I've removed one reference to that.
2. We're currently trying to encourage people to use the declarative setup.cfg file in favor of setup.py. It's much easier to reason about and it's easier to automatically pull metadata from without executing arbitrary code. Most of your metadata can easily be migrated, so I've done that, plus added a few other useful fields like python_requires.
3. Added a pyproject.toml file for PEP 517/518. This will change how pip builds your project, but it is intended to be backwards-compatible. This actually relies on some of the changes from the setup.cfg file, since when setup.py is invoked from a PEP 517 backend, it will no longer have the repository root on the PYTHONPATH. There's a simple workaround for this if you want to add a pyproject.toml file without migrating to setup.cfg.
4. I have added a build-and-check test to tox.ini. Right now you can only invoke it deliberately with tox -e build-check, but presumably you'll also want it to execute on CI and possibly add it to the default environments executed when you run tox. This relies on the existence of pyproject.toml, since pep517.build only works with PEP 517 builds.

[grantjenks]

I don’t really like these changes. But maybe that’s because I don’t understand them. Some questions:

1. What’s wrong with setup.py install?
2. Is keeping metadata in setup.py really so bad? setup.cfg and pyproject.toml just look like clutter. I don’t mind a little metadata in Python code.
3. What exactly does build-check do? I don’t get it.

I see changes but don’t get how they benefit me or others. Most people just install wheels. Does that change the wheel install process?

[pganssle]

Feel free to merge this anyway if you change your mind.

[pganssle]

What’s wrong with setup.py install?

This is, by the way, absolutely the wrong way to install Python packages. It is a decent way to mess up your environment, and for the past few years, if someone raised a bug with setup.py install and it isn't replicable with pip install, we mostly just close the bug.

The main reason we don't raise a warning on invocation is that it's tricky to avoid spurious warnings. At some point we'll take the time to get it right, then transition to removing the command.

[grantjenks]

@pganssle as part of V3 for sortedcontainers, I would like to merge these changes and modernize Python's build system. I would also like to publish cythonized wheels to PyPI. Using setup.cfg/pyproject.toml, is that possible? I would like to incorporate something like:

from Cython.Build import cythonize

setup(
    ...
    ext_modules=cythonize(...),
)

[pganssle]

I'm sorry, I have extremely limited time for this sort of thing, particularly since I don't directly use your project and really was only making this PR because it was easy for me and most people don't keep up with packaging.

• What’s wrong with setup.py install?

Many things. It is deprecated and you should not use it. Among the many, many reasons not to use setup.py, we have not been fixing anything in setuptools (or distutils) that is fixed by replacing setup.py install with pip install. I do not keep a tally of these things, but they are numerous.

• Is keeping metadata in setup.py really so bad? setup.cfg and pyproject.toml just look like clutter. I don’t mind a little metadata in Python code.

The declarative format is preferable for many reasons. One of which is that it's very easy for tools to tell which parts of the metadata are guaranteed to be static. Things are changing frequently, but we will likely

• What exactly does build-check do? I don’t get it.

I would think that's clear from the description: "Builds source and binary packages and uses twine check to validate". It's very common to make a mistake that doesn't break your tests but will break the build for whatever reason. This creates the build artifacts, then uses twine check to do whatever minimal checking is included in twine check (to make sure that the metadata doesn't break, for example). There are more exhaustive ways to test this sort of thing. I mainly included it so that I knew I wasn't obviously breaking the project with this. I don't really care if you use it.

I would also like to publish cythonized wheels to PyPI. Using setup.cfg/pyproject.toml, is that possible? I would like to incorporate something like:

You mean in the declarative metadata only? I'm not sure but at the moment I think you still need to put the cython step in setup.py. I don't see anything wrong with that, though, you can still get a pretty minimal setup.py. See, for example, this earlier version of zoneinfo (before I had to put in a bunch of nonsense to avoid warnings and gather coverage metrics properly).

[ml31415]

Sad to see, how even obvious improvements like this one, adaptions to recent python changes, stay unmerged for an eternity :(

[grantjenks]

@ml31415 thats not a motivating statement.

[ml31415]

@grantjenks it shouldn't be me needing to motivate you, but your own interest in keeping this very nice project up to date and relevant. There is a ton of a backlog of maintenance. I see lots of people trying to help out, but unfortunately an ugly lot of well intended PRs rejected and delayed. I have any understanding for a lack of time and not being able to work on things like that yourself. But in then you should just accept help from the ton of friendly programmers out there, that could help out with project maintenance.

[grantjenks]

@ml31415 thats not a motivating statement.

[pganssle]

FWIW, as the originator of this PR and as someone who has lately been terrible about my OSS backlog, I am not bothered by the fact that Grant hasn't merged this. I put it together because I appreciate the work he's done on this library and wanted to help out. The fact that I have offered my expertise on this topic does not put any onus on him to accept my changes.

Hopefully at some point this will be useful to Grant or someone else, but if not it's fine.

[pganssle]

Also, as an aside, this is also probably no longer the best way to do things. The conservative way to modernize the project would be to migrate from setup.py to pyproject.toml PEP 621 metadata (though I think there are tools that will automatically convert setup.cfg to pyproject.toml.

A lot has changed in packaging in the last few years, and there may also be better alternatives to setuptools even for C extensions (maybe hatchling?)

[grantjenks]

I've come around to pyproject.toml and have some changes locally that'll supersede these. Thanks @pganssle for the early exposure back in 2020.