pass multiple setup_args through pip install

[tacaswell]

I'm trying to build scipy on fedora which requires passing

python -m build --no-isolation --skip-dependency-check -Csetup-args=-Dblas=flexiblas  -Csetup-args=-Dlapack=flexiblas .

so that meson will correctly find lapack. This works because with build, the -C accumulate (observationally). In private communications with @rgommers he said that mulitple instances of -C was preferable to adding shlex style splitting to the strings passed to setup-args.

However it does not seem that --config-settings is pip does not accumulate and I can not find the right way to escape the command so that it flows through correctly (examples below). I am not sure if this is a issue in meson-python or should be reported upstream, but starting here.

---

Picks up the opening quote as part of the name

$ python -m pip install --no-build-isolation --config-settings="setup-args=-Dblas=flexiblas setup-args=-Dlapack=flexiblas" .
Processing /home/tcaswell/source/p/scipy/scipy
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [25 lines of output]
      Traceback (most recent call last):
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 144, in prepare_metadata_for_build_wheel
          hook = backend.prepare_metadata_for_build_wheel
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      AttributeError: module 'mesonpy' has no attribute 'prepare_metadata_for_build_wheel'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
          main()
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 148, in prepare_metadata_for_build_wheel
          whl_basename = backend.build_wheel(metadata_directory, config_settings)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/mesonpy/__init__.py", line 1070, in build_wheel
          with _project(config_settings) as project:
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.12/contextlib.py", line 137, in __enter__
          return next(self.gen)
                 ^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/mesonpy/__init__.py", line 980, in _project
          raise ConfigError(f'Unknown config setting: {key!r}.  {postfix}')
      mesonpy.ConfigError: Unknown config setting: '"setup-args'.  Did you mean one of: ['setup-args', 'dist-args', 'install-args']
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Does not split

$ python -m pip install --no-build-isolation "--config-settings=setup-args=-Dblas=flexiblas setup-args=-Dlapack=flexiblas" .
Processing /home/tcaswell/source/p/scipy/scipy
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [78 lines of output]
      + meson setup --prefix=/home/tcaswell/.pybuild/bleeding /home/tcaswell/source/p/scipy/scipy /home/tcaswell/source/p/scipy/scipy/.mesonpy-pj90uyro/build --native-file=/home/tcaswell/source/p/scipy/scipy/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2 -Dblas=flexiblas setup-args=-Dlapack=flexiblas
      The Meson build system
      Version: 0.64.0
      Source dir: /home/tcaswell/source/p/scipy/scipy
      Build dir: /home/tcaswell/source/p/scipy/scipy/.mesonpy-pj90uyro/build
      Build type: native build
      Project name: SciPy
      Project version: 1.10.0.dev0
      C compiler for the host machine: ccache cc (gcc 12.2.1 "cc (GCC) 12.2.1 20220819 (Red Hat 12.2.1-2)")
      C linker for the host machine: cc ld.bfd 2.38-25
      C++ compiler for the host machine: ccache c++ (gcc 12.2.1 "c++ (GCC) 12.2.1 20220819 (Red Hat 12.2.1-2)")
      C++ linker for the host machine: c++ ld.bfd 2.38-25
      Host machine cpu family: x86_64
      Host machine cpu: x86_64
      Compiler for C supports arguments -Wno-unused-but-set-variable: YES
      Compiler for C supports arguments -Wno-unused-function: YES
      Compiler for C supports arguments -Wno-conversion: YES
      Compiler for C supports arguments -Wno-misleading-indentation: YES
      Compiler for C supports arguments -Wno-incompatible-pointer-types: YES
      Library m found: YES
      Fortran compiler for the host machine: gfortran (gcc 12.2.1 "GNU Fortran (GCC) 12.2.1 20220819 (Red Hat 12.2.1-2)")
      Fortran linker for the host machine: gfortran ld.bfd 2.38-25
      Compiler for Fortran supports arguments -Wno-conversion: YES
      Checking if "-Wl,--version-script" : links: YES
      Program cython found: YES (/home/tcaswell/.virtualenvs/bleeding/bin/cython)
      Program python found: YES (/home/tcaswell/.virtualenvs/bleeding/bin/python)
      Found pkg-config: /usr/bin/pkg-config (1.8.0)
      Program pythran found: YES (/home/tcaswell/.virtualenvs/bleeding/bin/pythran)
      Run-time dependency threads found: YES
      Library npymath found: YES
      Library npyrandom found: YES
      
      ../../scipy/meson.build:134:0: ERROR: Characters <, > and = are forbidden in dependency names. To specifyversion
       requirements use the 'version' keyword argument instead.
      
      A full log can be found at /home/tcaswell/source/p/scipy/scipy/.mesonpy-pj90uyro/build/meson-logs/meson-log.txt
      Traceback (most recent call last):
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 144, in prepare_metadata_for_build_wheel
          hook = backend.prepare_metadata_for_build_wheel
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      AttributeError: module 'mesonpy' has no attribute 'prepare_metadata_for_build_wheel'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
          main()
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 148, in prepare_metadata_for_build_wheel
          whl_basename = backend.build_wheel(metadata_directory, config_settings)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/mesonpy/__init__.py", line 1067, in build_wheel
          with _project(config_settings) as project:
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.12/contextlib.py", line 137, in __enter__
          return next(self.gen)
                 ^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/mesonpy/__init__.py", line 982, in _project
          with Project.with_temp_working_dir(
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.12/contextlib.py", line 137, in __enter__
          return next(self.gen)
                 ^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/mesonpy/__init__.py", line 750, in with_temp_working_dir
          yield cls(source_dir, tmpdir, build_dir, meson_args)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/mesonpy/__init__.py", line 632, in __init__
          self._configure(reconfigure=bool(build_dir) and not native_file_mismatch)
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/mesonpy/__init__.py", line 680, in _configure
          self._meson('setup', *setup_args)
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/mesonpy/__init__.py", line 657, in _meson
          return self._proc('meson', *args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/mesonpy/__init__.py", line 652, in _proc
          subprocess.check_call(list(args), env=self._env)
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.12/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['meson', 'setup', '--prefix=/home/tcaswell/.pybuild/bleeding', '/home/tcaswell/source/p/scipy/scipy', '/home/tcaswell/source/p/scipy/scipy/.mesonpy-pj90uyro/build', '--native-file=/home/tcaswell/source/p/scipy/scipy/.mesonpy-native-file.ini', '-Ddebug=false', '-Doptimization=2', '-Dblas=flexiblas setup-args=-Dlapack=flexiblas']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

Only keeps the last flag:

$ python -m pip install --no-build-isolation --config-settings=setup-args=-Dblas=flexiblas --config-settings=setup-args=-Dlapack=flexiblas .
Processing /home/tcaswell/source/p/scipy/scipy
  Preparing metadata (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Preparing metadata (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [80 lines of output]
      + meson setup --prefix=/home/tcaswell/.pybuild/bleeding /home/tcaswell/source/p/scipy/scipy /home/tcaswell/source/p/scipy/scipy/.mesonpy-3zs0lpwm/build --native-file=/home/tcaswell/source/p/scipy/scipy/.mesonpy-native-file.ini -Ddebug=false -Doptimization=2 -Dlapack=flexiblas
      The Meson build system
      Version: 0.64.0
      Source dir: /home/tcaswell/source/p/scipy/scipy
      Build dir: /home/tcaswell/source/p/scipy/scipy/.mesonpy-3zs0lpwm/build
      Build type: native build
      Project name: SciPy
      Project version: 1.10.0.dev0
      C compiler for the host machine: ccache cc (gcc 12.2.1 "cc (GCC) 12.2.1 20220819 (Red Hat 12.2.1-2)")
      C linker for the host machine: cc ld.bfd 2.38-25
      C++ compiler for the host machine: ccache c++ (gcc 12.2.1 "c++ (GCC) 12.2.1 20220819 (Red Hat 12.2.1-2)")
      C++ linker for the host machine: c++ ld.bfd 2.38-25
      Host machine cpu family: x86_64
      Host machine cpu: x86_64
      Compiler for C supports arguments -Wno-unused-but-set-variable: YES
      Compiler for C supports arguments -Wno-unused-function: YES
      Compiler for C supports arguments -Wno-conversion: YES
      Compiler for C supports arguments -Wno-misleading-indentation: YES
      Compiler for C supports arguments -Wno-incompatible-pointer-types: YES
      Library m found: YES
      Fortran compiler for the host machine: gfortran (gcc 12.2.1 "GNU Fortran (GCC) 12.2.1 20220819 (Red Hat 12.2.1-2)")
      Fortran linker for the host machine: gfortran ld.bfd 2.38-25
      Compiler for Fortran supports arguments -Wno-conversion: YES
      Checking if "-Wl,--version-script" : links: YES
      Program cython found: YES (/home/tcaswell/.virtualenvs/bleeding/bin/cython)
      Program python found: YES (/home/tcaswell/.virtualenvs/bleeding/bin/python)
      Found pkg-config: /usr/bin/pkg-config (1.8.0)
      Program pythran found: YES (/home/tcaswell/.virtualenvs/bleeding/bin/pythran)
      Run-time dependency threads found: YES
      Library npymath found: YES
      Library npyrandom found: YES
      Found CMake: /home/tcaswell/.virtualenvs/bleeding/bin/cmake (3.25.0)
      Run-time dependency openblas found: NO (tried pkgconfig and cmake)
      Run-time dependency openblas found: NO (tried pkgconfig and cmake)
      
      ../../scipy/meson.build:134:0: ERROR: Dependency "OpenBLAS" not found, tried pkgconfig and cmake
      
      A full log can be found at /home/tcaswell/source/p/scipy/scipy/.mesonpy-3zs0lpwm/build/meson-logs/meson-log.txt
      Traceback (most recent call last):
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 144, in prepare_metadata_for_build_wheel
          hook = backend.prepare_metadata_for_build_wheel
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      AttributeError: module 'mesonpy' has no attribute 'prepare_metadata_for_build_wheel'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 351, in <module>
          main()
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 333, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 148, in prepare_metadata_for_build_wheel
          whl_basename = backend.build_wheel(metadata_directory, config_settings)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/mesonpy/__init__.py", line 1067, in build_wheel
          with _project(config_settings) as project:
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.12/contextlib.py", line 137, in __enter__
          return next(self.gen)
                 ^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/mesonpy/__init__.py", line 982, in _project
          with Project.with_temp_working_dir(
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.12/contextlib.py", line 137, in __enter__
          return next(self.gen)
                 ^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/mesonpy/__init__.py", line 750, in with_temp_working_dir
          yield cls(source_dir, tmpdir, build_dir, meson_args)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/mesonpy/__init__.py", line 632, in __init__
          self._configure(reconfigure=bool(build_dir) and not native_file_mismatch)
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/mesonpy/__init__.py", line 680, in _configure
          self._meson('setup', *setup_args)
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/mesonpy/__init__.py", line 657, in _meson
          return self._proc('meson', *args)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/tcaswell/.virtualenvs/bleeding/lib/python3.12/site-packages/mesonpy/__init__.py", line 652, in _proc
          subprocess.check_call(list(args), env=self._env)
        File "/home/tcaswell/.pybuild/bleeding/lib/python3.12/subprocess.py", line 413, in check_call
          raise CalledProcessError(retcode, cmd)
      subprocess.CalledProcessError: Command '['meson', 'setup', '--prefix=/home/tcaswell/.pybuild/bleeding', '/home/tcaswell/source/p/scipy/scipy', '/home/tcaswell/source/p/scipy/scipy/.mesonpy-3zs0lpwm/build', '--native-file=/home/tcaswell/source/p/scipy/scipy/.mesonpy-native-file.ini', '-Ddebug=false', '-Doptimization=2', '-Dlapack=flexiblas']' returned non-zero exit status 1.
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: metadata-generation-failed

× Encountered error while generating package metadata.
╰─> See above for output.

note: This is an issue with the package mentioned above, not pip.
hint: See above for details.

[dnicolodi]

No amount of quoting will never split the arguments correctly to be passed to meson: there is no lexing done in meson-python and passing them as a single argument is not going to work.

The only place where I see the command line argument handling for pep517 frontends specified is here https://peps.python.org/pep-0517/#config-settings and the example makes it clear that passing the same option multiple times should be supported. Also, build allows repeating options but pip does not, thus they require different handling that cannot be reconciled to work with both.

I think this is a pip bug. Fortunately it can be worked-around easily: build a wheel with build and install it with pip.

[dnicolodi]

Even pip's own documentation states that --config-setting may be specified mutliple times:

$ python -m pip install --help
...
  --config-settings <settings>
                              Configuration settings to be passed to the PEP
                              517 build backend. Settings take the form
                              KEY=VALUE. Use multiple --config-settings
                              options to pass multiple keys to the backend.
...

[henryiii]

Pip does not support merging x=a x=b into x=[a,b] like build does, and never has. See https://github.com/pypa/cibuildwheel/issues/1227#issuecomment-1212972828. It does support multi arguments if they are not the same. It also doesn't support -C, which I would love for it to do, partially because -Cx=a looks better than --config-settings=x=a, and partially because that would provide a uniform interface for pip and build, since build calls it --config-setting instead.

[rgommers]

Pip does not support merging x=a x=b into x=[a,b] like build does, and never has.

Now it does, after https://github.com/pypa/pip/pull/11853 🎉. This will start working with the next pip release, so let's close this issue.