Deprecate `--global-option` and `--build-option`

[sbidoul]

The use of --global-option and --build-option currently trigger a warning, recommending --config-settings.

I propose to deprecate them with a 6 month notice and remove them in 23.3.

After that, we can consider dropping the legacy setup.py egg_info + setup.py bdist_wheel code paths and switch to using the default build backend ~without build isolation~ for projects without pyproject.toml.

[pradyunsg]

Alternatively, we could forward those options to backends on a pyproject.toml based build, via config settings?

[uranusjr]

Realistically the option translation is only useful for projects using setuptools, so we should first ask whether setuptools devs are interested in establishing such translation.

[sbidoul]

Alternatively, we could forward those options to backends on a pyproject.toml based build, via config settings?

I guess it's possible in principle, although I can imagine some difficulties (e.g. does setuptools distinguish global and build options in config settings?).

This would also likely trigger requests for adding --build-option to pip install, since it's currently in pip wheel only.

So, all in all, I think it would be quite a bit of fuss for little benefits. Especially since we'll likely want to drop these setuptools specific options at some point in any case.

[pradyunsg]

These are both referenced in the setuptools implementation today: https://github.com/pypa/setuptools/blob/be6c0218bcba78dbd4ea0b5a8bb9acd5d5306240/setuptools/build_meta.py#L158

As far as I can remember, these have been implemented from fairly early on in the build-backend provided by setuptools.

does setuptools distinguish global and build options in config settings?

Yes.

This would also likely trigger requests for adding --build-option to pip install, since it's currently in pip wheel only.

Sure, and that's a valid request, no? It pushes for consistency, and I'm ~sure that we can leverage that consistency to make the codebase clearer around this aspect as well.

So, all in all, I think it would be quite a bit of fuss for little benefits.

I don't think so. Something that we spent a lot of energy toward ensuring was that we're able to transition users toward the newer build mechanism without being too disruptive to their workflow, outside of certain bits of functionality that don't have equivalents that are being removed (and that goal hitting Hyrum's law).

However, this is functionality that we can implement in a manner that avoids disruption. The PEP even has an example of the behaviour I'm proposing in the config settings section (https://peps.python.org/pep-0517/#config-settings).

[pfmoore]

I can see the point of translating these options to config settings. But I think we should consider the larger question, do we want to retain these options forever, or not? There are arguments both ways:

1. (Keep) People are (presumably) using these right now, so we'd break workflows by removing them.
2. (Remove) There's a standards-based replacement which is already supported, so migration is straightforward.
3. (?) The existing implementation is incomplete (--build-option on install). We can, of course, complete it.
4. (Remove) The flags add a fair amount of complexity (command line options, requirements file support, threading the flags through the various call chains, etc.)
5. (Remove) The flags are completely setuptools-specific, and we have a stated goal of removing all special consideration of setuptools.
6. (Keep) The PEP calls this example out explicitly (although that section is only an example).

Personally, I find the arguments for removal compelling, but not urgent. The 6-month deprecation seems like a reasonable compromise, but I'd be fine with extending it further. I'm not convinced there's a case for retaining the options forever (and by implication, I'm against "fixing" the problem of install not having a --build-option flag).

For me, the main point is that this is another "remove special-casing of setuptools" change, and doing it in parallel with the removal of the setup.py call logic makes sense, both from an internal POV, and in terms of user disruption (having a single period of "removing setuptools-specific features" disruption for users, is arguably better than having a series of separate disruptions).

[sbidoul]

For me, the main point is that this is another "remove special-casing of setuptools" change, and doing it in parallel with the removal of the setup.py call logic makes sense, both from an internal POV, and in terms of user disruption (having a single period of "removing setuptools-specific features" disruption for users, is arguably better than having a series of separate disruptions).

This is also my point of view. In particular, since users still relying on --install-option must switch to something else with 23.1 it is better to state clearly right now that --global-option is not going to be around forever so they don't settle on that to discover in one year that we want to remove them too.

[pradyunsg]

Fair enough! Color me convinced that we should deprecate + remove these. :)

[sbidoul]

Merged in 23.1. Moving this issue to 23.3 for removal.

[beckermr]

Am I following correctly that unless this setuptools issue (https://github.com/pypa/setuptools/issues/2491) is addressed, once pip 23.3 lands there will be no way to pass extra options through pip to setuptools?

[pfmoore]

@beckermr Maybe (I don't pretend to understand what setuptools is doing here) - you should probably ask on the linked setuptools issue as well, and make them aware of the urgency.

[alexchandel]

Is there an example showing how to use --config-settings with setup.py and/or newer alternatives? The setuptools documentation is awful and the top search results are years/decades out-of-date and wildly contradictory.

[sbidoul]

Is there an example showing how to use --config-settings with setup.py and/or newer alternatives? The setuptools documentation is awful and the top search results are years/decades out-of-date and wildly contradictory.

Something like this seems to work to pass global options to setuptools.

pip -vv install --config-setting="--global-option=--verbose" .

Passing --build-option in the same way does not work, as setuptools attempts to pass these to the egg_info command where they are not supported.

[sbidoul]

https://github.com/pypa/setuptools/issues/3896 seems to be the a current discussion about config settings support in setuptools, with various workaround being proposed.

[indygreg]

I discovered this issue after spending several hours attempting to port my python setup.py invocations to python -m pip ... (my setup.py scans sys.argv looking for custom arguments).

As others have noted, the setuptools documentation on how to pass custom config options / process arguments to setuptools is pretty much non-existent. However, it is possible. (I had to look at the source code to understand what's going on.)

Older versions of setuptools support passing additional arguments via pip via --config-setting='--global-option=...'. However, this is a bit hacky because these are global arguments passed to every setuptools command. That's probably not what you want! And use of --config-setting='--global-option=...' may yield a deprecation warning in setuptools.

setuptools 64.0.0+ supports a --build-option config setting to pass arguments to just the build commands. e.g. from pip --config-setting='--build-option=--my-custom-option'. This seems to work. However, the feature isn't documented outside the setuptools changelog (https://github.com/pypa/setuptools/blob/main/NEWS.rst#v6400). And that same changelog says the feature is only transitional, implying it won't be supported forever. https://github.com/pypa/setuptools/issues/3896 seems to be the discussion around what the eventual long-term feature/support for passing custom config settings will look like.

setuptools 64.0.0+ also seems to support preserving arbitrary, unrecognized arguments. e.g. --config-setting='--my-custom-argument' (See code at https://github.com/pypa/setuptools/blob/2384d915088b960999ca74fb81ce70bffd17b082/setuptools/build_meta.py#L269). However, this raises a deprecation warning today. And the arguments are passed to all commands. So this feature has the same caveats as --global-option`.

It seems that at this time setuptools hasn't really committed to a stable interface for passing arbitrary config settings to setup.py arguments short of implementing your own build backend (https://setuptools.pypa.io/en/latest/build_meta.html#dynamic-build-dependencies-and-other-build-meta-tweaks).

I suppose I'm fine with custom build backends being the long term solution for us power users wanting to bring our own build config settings. However, the existing documentation on how to do all of this is pretty much non-existent. I couldn't find any documentation under the PyPA umbrella on how to migrate existing setup.py code and invocations to the new desired mechanism. It isn't clear to me if this is even safe to do: could I break legacy python setup.py invocations if they aren't picking up modern setuptools versions via a pyproject.toml [build-system] version requirement or are running with an older pip?

I want to support the transition to modern build system abstractions. But it seems to me that the interface between setuptools and build frontends (like pip) is still not clearly defined. I think most of the blame here is on setuptools since they haven't provided documented/supported on ramps to the new world order. But the reality is that the average user doesn't understand the boundaries between build frontends (like pip) and setuptools and they are just going to blame pip when setup.py based builds - which in many cases have been working for 10+ years with minimal modifications - suddenly require new, awkward incantations to invoke (--config-setting='--build-option=--my-custom-argument' is really convoluted) and start spewing deprecation warnings.

Given the popularity and historical stability of setuptools, I'd urge the pip maintainers to exercise restraint in deprecating --global-option and --build-option until after setuptools has committed to a stable, documented mechanism for handling custom config settings. From my armchair perspective as a package maintainer, it isn't clear what the endorsed path forward is because setuptools nor the larger PyPA ecosystem have provided well documented guidance. It seems premature to deprecate working, simpler mechanisms (--global-option / --build-option are simpler than --config-settings='--build-option=...'). Especially if a future setuptools release deprecates its transitional --build-option in favor of something else. As a package maintainer, I strongly prefer to perform this build system packaging migration once. If pip deprecates --global-option/--build-option we run the risk of forcing multiple migrations onto package maintainers.

[flying-sheep]

I suppose I'm fine with custom build backends being the long term solution for us power users wanting to bring our own build config settings.

I hope Ofek’s vision comes true, and there doesn‘t need to be one build backend for each kind of extension module that can be built. Why should maturin or pybind11 concern themselves with building a wheel with correct metadata?

[sbidoul]

Thanks for your feedback, @indygreg.

The pip deprecation process is indeed the only practical way we have to collect user feedback and gather information to help us decide how and when to remove legacy features [^1]. So your input is very valuable.

[^1]: It also helps nudging the ecosystem towards modern standards, as without the pip warnings, nobody would notice and there would be little incentive for change. And change is necessary. We do take backward compatibility very seriously but at some point supporting legacy gets too much in the way of evolution.

[eli-schwartz]

@flying-sheep

Since pybind11 is a C++ header only library, I think you're asking the wrong question.

The question isn't "why should pybind11 know how to generate wheel metadata", the question is "why should a collection of headers know anything other than how to be a collection of headers"? One should not be building extension modules that use pybind11 using a build system shipped with pybind11 -- one should be using setuptools, meson, scikit-build, or similar. Software that specializes in compiling C/C++ software, maybe limited to a python context but also maybe generic for the whole wide world of software.

Maturin is interesting because it is, in fact, a build system and compiles python extension modules. (It has to be because there are a very small number of build systems for rust: cargo, meson, bazel. And maturin wraps cargo for you and adds knowledge of python.) I don't see a compelling argument against it building wheels too, especially since the code to do so already exists, and it allows installing one thing rather than 4 or 5 things.

---

There are much better examples in the "food for thought" arena than either of these two which exist at opposite ends of a spectrum and have easy answers. How about a shade of grey?

Consider mypyc and cython. They are code generators which emit C(++) code. They are also wrappers around setuptools that build setuptools Extensions for you. Should they include the latter, or not? If not, how easy is it to write a rule inside an independent build system to do it for you? (In the case of mypy it seems pretty hard, actually.) And since both provide actual honest to goodness CLIs, whereas setuptools has deprecated its CLI... how stable or reliable is it to build your own build system by subclassing setuptools?

I guess that Ofek would say the answer is they should include code to build extension modules, but not use setuptools for it and not bother with wheels. My personal take on the matter is that they shouldn't have any code whatsoever to build extensions, but make it easy to be called from external build systems such as meson and cmake. Actually, this is already the case for cython.

[eli-schwartz]

Anyways, the point is moot since there aren't multiple types of extensions here as far as I can tell -- just the need to pass options to setup.py without them being incorrectly parsed by setuptools.build_meta.

Everything else is happening in stock setup.py -- unless the answer is that setuptools doesn't make wheels either, just builds C extensions and passes them to hatchling to be made into a wheel.

I think that will be a hard sell to most stakeholders.

[webknjaz]

@sbidoul does --config-settings get passed to all the packages? The deprecated --install-option used to be per-package and didn't leak into others: https://github.com/pypa/pip/pull/11858/files#diff-ae0a88ffece2e628ec4927675709082e92c646944a612c72432480c0e7dffb8dL768.

[sbidoul]

does --config-settings get passed to all the packages

@webknjaz according to https://github.com/pypa/pip/pull/11941 and the linked comment there, it should not.

[sbidoul]

Postponing this to no earlier than pip 24.2.

[indygreg]

A few days after I wrote my long comment in October, setuptools merged https://github.com/pypa/setuptools/pull/4079 (first shipped in 69.0.0) which only passes --config-settings=--build-option to the bdist_wheel command. This effectively broke workflows. I'm pretty sure it completely broke --config-settings for editable installs, as my CI using pip install --config-settings='--build-option=...' -e . breaks on setuptools >= 69.0.0.

There's various discussion of this at https://github.com/pypa/setuptools/pull/4218, https://github.com/pypa/setuptools/pull/4217, https://github.com/pypa/setuptools/issues/2491, and https://github.com/pypa/setuptools/discussions/4083.

setuptools still doesn't have a documented nor stable solution for threading --config-settings through to their PEP 517 build backend. And since I last commented they actually made it harder to pass custom build options.

I want to reaffirm my earlier position that it still feels premature to deprecate anything here until setuptools has a stable, documented solution for supporting dynamic configuration via --config-settings.