Setuptools does not pass config_settings through backend

[di]

The build project now has a --config-setting flag to pass options to the backend:

$ python -m build --help
usage: python -m build [-h] [--version] [--sdist] [--wheel] [--outdir dir] [--skip-dependencies]
                       [--no-isolation] [--config-setting CONFIG_SETTING]
                       [srcdir]

- - - >8 - - -

  --config-setting CONFIG_SETTING, -C CONFIG_SETTING
                        pass option to the backend

I wanted to use this to set wheel's --plat-name argument like so:

$ python -m build --wheel --config-setting plat-name=foo

I found that this worked as expected until the settings got to setuptool's _build_with_temp_dir function:

https://github.com/pypa/setuptools/blob/d368be2b5be8676ba8a3d55045b45b55090f6982/setuptools/build_meta.py#L190-L200

In this example, _build_with_temp_dir gets called with config_settings={'plat-name': 'foo'}, which then gets 'fixed' into config_settings={'plat-name': 'foo', '--global-option': []}, and then setuptools ignores everything in config_settings and only considers --global-option.

It almost seems like --global-option itself could be used for this, but unfortunately it doesn't work as it's value is passed to lots of things that don't accept the arguments I'm hoping to pass:

$ python -m build --wheel --config-setting="--global-option=--plat-name" --config-setting="--global-option=foo" -n
usage: _in_process.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: _in_process.py --help [cmd1 cmd2 ...]
   or: _in_process.py --help-commands
   or: _in_process.py cmd --help

error: option --plat-name not recognized
...

I created a commit that "fixes" this for me as a proof of concept: https://github.com/pypa/setuptools/commit/fc95b3b83d6d5b561dc0a356995edf4c99785a6f

Possibly related issues:

• this is very similar to the request in https://github.com/pypa/setuptools/issues/1816, but @gaborbernat is asking for the ability to pass options before the setup command, rather than after.
• https://github.com/pypa/setuptools/issues/1928 may also be related

[jaraco]

I could have used something like this too for Setuptools itself to disable post-release tags for #2500 whereas currently it's using this hack. I suspect Setuptools needs to devise a way to map "config settings" to tweak knobs at different scopes in the build process (global, per-command, ...).

[abitrolly]

@di why not put this into PR?

[di]

@abitrolly Because I'm not sure it's right. The means by which a frontend passes these arguments to the backend is not well defined.

[abitrolly]

@di what is missing here https://www.python.org/dev/peps/pep-0517/#config-settings ?

[di]

@abitrolly All these phrases from that section indicate that this is not well defined:

Build backends MAY assign any semantics they like to this dictionary.

Build frontends SHOULD provide some mechanism for users to specify arbitrary string-key/string-value pairs to be placed in this dictionary.

Build frontends MAY also provide arbitrary other mechanisms for users to place entries in this dictionary

[abitrolly]

@di I proposed some clarifications in https://github.com/python/peps/pull/1766 Not sure where it should go from there.

Normalizing parameters to be passed as "option": "value" without dashes seems to be a good compromise. Because interface to build system is defined as callback functions and not as CLI API, keeping dashes would mean that backend should somehow pass the names through argparse.

[FFY00]

FYI I am also in favor of removing the dashes, I don't think they are needed in PEP 517.

[layday]

Incidentally, global options in setuptools seem to mean something different than they do in pip where they are placed in front of the setuptools command. Here, they are appended to the command. pip users might find the discrepancy surprising.

I'm not sure why arguments to sdist and bdist_wheel should not be funneled through config_settings verbatim - why is --global-option needed at all?

[jaraco]

I'm sure the reason why the PEP is non-specific about the format is because it would be difficult for a protocol like that to be specific without being over-constrained for a particular use-case.

I'm not sure why arguments to sdist and bdist_wheel should not be funneled through config_settings verbatim - why is --global-option needed at all?

I imagine that the sdist and bdist_wheel commands have settings that are unique to the command. I also expect there could be settings that are applied globally, independent of any command.

Regardless, what I'd like to see happen is for someone to examine the code and determine all the command-line parameters setuptools currently solicits that would be relevant to a build backend... and determine at what scopes those settings are relevant (globally, per command, other?). From there, we can create a scheme, something simple, maybe namespaced, to accept those parameters through config_settings.

[layday]

Issues raised against build concerning this behaviour:

• https://github.com/pypa/build/issues/202#issuecomment-757373572
• https://github.com/pypa/build/issues/258
• https://github.com/pypa/build/issues/264
• https://github.com/pypa/build/issues/328#issuecomment-877028239
• https://github.com/pypa/build/issues/417
• https://github.com/pypa/build/issues/421
• https://github.com/pypa/build/issues/432

[abitrolly]

@layday if you want to push this further, I guess the right way it to do what @brettcannon said in https://github.com/python/peps/pull/1766#issuecomment-762487258

Please discuss any proposed changes at https://discuss.python.org/c/packaging/14 first.

[layday]

I don't know how changing lists to comma-separated value strings in the PEP is going to help here.

[abitrolly]

@layday because PEP doesn't specify how to pass lists, nobody wants to write this.

[dixonwhitmire]

Hi @abitrolly so just to clarify for myself and possibly for others following this issue, is the next step here to discuss potential changes in https://discuss.python.org/c/packaging/14 so that the PEP is updated with well defined statements? Thank you!

[abitrolly]

@dixonwhitmire yes, that's correct.

[PolyacovYury]

Just encountered this. My use-case was to pass --quiet from python -m build call.

First there is an issue with build itself: passing -C--global-option=--quiet once causes an error, since config_settings['--global-option'] becomes a string, and setuptools expects a list. Maybe change _fix_config() to something like:

    def _fix_config(self, config_settings):
        config_settings = config_settings or {}
        config_settings.setdefault('--global-option', [])
        if not isinstance(config_settings['--global-option'], list):
            config_settings['--global-option'] = [config_settings['--global-option']]
        return config_settings

Second issue is that setuptools does NOT conform to https://www.python.org/dev/peps/pep-0517/#config-settings. --global-option should be inserted in front of the setup_command, and --build-option should go to the end of the argv. This way, invoking python -m build -C--global-option=--quiet will produce the expected result.

Although, even better would be to adjust build itself, so that instead of -C it has -G and -B (for --global-option and --build-option respectively), but that would be out of scope of this discussion.

[abravalheri]

Hi @PolyacovYury , I think the first part of the problem you described was solved in https://github.com/pypa/setuptools/pull/2876, wasn't it?

Regarding the second part:

--global-option should be inserted in front of the setup_command

It is not immediately clear to me that the spec mandates that --global-option should come before setup_command... Is there any specific part of the document or is this something you concluded is necessary for --quiet to work? (I wonder it this order, while working for --quiet might break other options that specific to the sdist or bdist_wheel subcommands).

[layday]

I commented on this upthread:

Incidentally, global options in setuptools seem to mean something different than they do in pip where they are placed in front of the setuptools command. Here, they are appended to the command. pip users might find the discrepancy surprising.

---

It is not immediately clear to me that the spec mandates that --global-option should come before setup_command [...]

This is all setuptools-specific stuff, the spec has no notion of global options nor does it expect that options will be passed to a CLI.

[PolyacovYury]

I think the first part of the problem you described was solved in https://github.com/pypa/setuptools/pull/2876, wasn't it?

Looks like it was, yes. Might need to update my venvs then, but when I tested this (the day before the comment was posted) this was either not published to PyPI or somehow managed to not get pulled by my pip. Me still having python 3.6 is to blame, perhaps?..

---

It is not immediately clear to me that the spec mandates that --global-option should come before setup_command... Is there any specific part of the document...?

Oops. Was kinda frustrated after a 2-day-long debug session, which resulted in me not checking the docs thoroughly enough.
@layday is right:

This is all setuptools-specific stuff.

---

the spec has no notion of global options nor does it expect that options will be passed to a CLI.

Despite that, having the config_options working consistently across tools...

global options in setuptools seem to mean something different than they do in pip where they are placed in front of the setuptools command. Here, they are appended to the command. pip users might find the discrepancy surprising.

...would be very helpful. Also, having --global-option affect the commands rather than the CLI as a whole makes my brain hurt while trying to come up with a name for an option which would affect the CLI. --dist-option? --setup-option?

---

is this something you concluded is necessary for --quiet to work?

Yes. As well as for any other command-line options defined in the distutils/dist.py:Distribuiton class's global_options list.

---

I wonder it this order, while working for --quiet, might break other options that specific to the sdist or bdist_wheel subcommands.

distutils/dist.py:Distribuiton's first call to FancyGetopt.getopt in its parse_command_line() pulls out arguments given before the first build command and applies them to the Distribution class itself (parser.getopt(args=self.script_args, object=self)), thus these arguments would never be passed to the actual build command. The args list returned by that call always starts with the first build command.

So, for example, if build_wheel had a --quiet option that would do something different from the global one, and I wanted the behaviours of both global and command options, I would need to invoke the CLI as setup.py --quiet bdist_wheel --quiet.
Running setup.py --quiet bdist_wheel results in bdist_wheel not even knowing that --quiet was ever there.

Incidentally, invoking setup.py bdist_wheel --quiet results in Distribution never getting the memo about quietness. Which is the reason of my arrival in this thread in the first place :)

As for current build commands' implementation's reaction to --quiet/--verbose... distutils/cmd.py:Command class's __init__ has you covered:

# verbose is largely ignored, but needs to be set for
# backwards compatibility (I think)?
self.verbose = dist.verbose

...and self.verbose is never referred to. Ever. Even distutils/ccompiler.py:new_compiler() ultimately does nothing with it.

[abravalheri]

This might be relevant for this thread: I believe that in v64.0.3+ setuptools is using config_settings["--global-option"] and config_settings["--build-option"] consistently with what used to happen in pip.

[ssbarnea]

@abravalheri Something is not right...

.package installdeps: setuptools >= 64.0.3, setuptools_scm[toml] >= 3.5.0, setuptools_scm_git_archive >= 1.0, wheel
/Users/ssbarnea/c/os/tendo/.tox/.package/lib/python3.11/site-packages/setuptools/build_meta.py:307: SetuptoolsDeprecationWarning: 
            The arguments ['--formats=gztar'] were given via `--global-option`.
            Please use `--build-option` instead,
            `--global-option` is reserved to flags like `--verbose` or `--quiet`.
            
  warnings.warn(msg, SetuptoolsDeprecationWarning)

[abravalheri]

Hi @ssbarnea , this behaviour is consistent with https://github.com/pypa/setuptools/issues/1928 isn't it?

[alex]

I am not sure if this is related, but when attempting to use --config-settings to pass py-limited-api, I'm getting the error:

  usage: _in_process.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: _in_process.py --help [cmd1 cmd2 ...]
     or: _in_process.py --help-commands
     or: _in_process.py cmd --help

  error: option --py-limited-api not recognized

[layday]

It's probably 'cause pip is calling [setuptools.build_meta.]get_requires_for_build_wheel to ensure all build dependencies are satisfied prior to building the wheel, which in turn calls setup.py egg_info with your --config-settings, for which --py-limited-api=... is an invalid option. (If you pass --no-build-isolation to pip wheel, it won't call get_requires_for_build_wheel, but that does mean you'll have to install your build dependencies separately.) setuptools doesn't have a way to sift out build options by command.

[hugovk]

We're considering something like https://github.com/python-pillow/Pillow/pull/7171 to work around this with a custom backend. Does anyone have a nicer way to do it?

[FFY00]

@hugovk IMO that is a very reasonable way to do it, and would also be my recommendation.

[di]

Interesting workaround!

[sbidoul]

This might be relevant for this thread: I believe that in v64.0.3+ setuptools is using config_settings["--global-option"] and config_settings["--build-option"] consistently with what used to happen in pip.

@abravalheri I'm preparing to remove --global-option and --build-option from pip and trying to pass options via config settings.

--global-option seems to work fine.

But I notice that the --build-option makes the setuptools backend fail. It looks like it tries to pass the build options to the egg_info command:

$ py -m pip -vv install --use-pep517  --config-setting="--global-option=--verbose" --config-setting="--global-option=--quiet" --config-setting="--build-option=--universal" .
Using pip 23.3.dev0 from /home/sbi-local/FOSS/pip/src/pip (python 3.8)
Non-user install because user site-packages disabled
Created temporary directory: /tmp/pip-build-tracker-t9g9dhw_
Initialized build tracking at /tmp/pip-build-tracker-t9g9dhw_
Created build tracker: /tmp/pip-build-tracker-t9g9dhw_
Entered build tracker: /tmp/pip-build-tracker-t9g9dhw_
Created temporary directory: /tmp/pip-install-lzd7kyz5
Created temporary directory: /tmp/pip-ephem-wheel-cache-qfdbhpi0
Processing /tmp/brol
  Added file:///tmp/brol to build tracker '/tmp/pip-build-tracker-t9g9dhw_'
  Created temporary directory: /tmp/pip-build-env-71nero8h
  Running command pip subprocess to install build dependencies
  Using pip 23.3.dev0 from /home/sbi-local/FOSS/pip/src/pip (python 3.8)
  Collecting setuptools>=40.8.0
    Obtaining dependency information for setuptools>=40.8.0 from https://files.pythonhosted.org/packages/bb/26/7945080113158354380a12ce26873dd6c1ebd88d47f5bc24e2c5bb38c16a/setuptools-68.2.2-py3-none-any.whl.metadata
    Using cached setuptools-68.2.2-py3-none-any.whl.metadata (6.3 kB)
  Collecting wheel
    Obtaining dependency information for wheel from https://files.pythonhosted.org/packages/b8/8b/31273bf66016be6ad22bb7345c37ff350276cfd46e389a0c2ac5da9d9073/wheel-0.41.2-py3-none-any.whl.metadata
    Using cached wheel-0.41.2-py3-none-any.whl.metadata (2.2 kB)
  Using cached setuptools-68.2.2-py3-none-any.whl (807 kB)
  Using cached wheel-0.41.2-py3-none-any.whl (64 kB)
  Installing collected packages: wheel, setuptools
    Creating /tmp/pip-build-env-71nero8h/overlay/bin
    changing mode of /tmp/pip-build-env-71nero8h/overlay/bin/wheel to 775
  Successfully installed setuptools-68.2.2 wheel-0.41.2
  Installing build dependencies ... done
  Running command Getting requirements to build wheel
  ************ ['setup.py', '--verbose', '--quiet', 'egg_info', '--universal']
  usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: setup.py --help [cmd1 cmd2 ...]
     or: setup.py --help-commands
     or: setup.py cmd --help

  error: option --universal not recognized
  error: subprocess-exited-with-error

[sbidoul]

I confirm the same error occurs with the build frontend.

[sbidoul]

I just realize this is discussed at length in https://github.com/pypa/setuptools/issues/3896 too.