Unify the shells we use for before-build, test-command etc?

[joerick]

Thinking about #1779, I'm always on the lookout for a distributable sh/bash implementation on windows, it would be really nice to unify this one day. Advantages would be

• shared syntax, e.g. env vars are always accessed with $VAR, command substitution
• less need for platform-specific configs, some commands can be shared
• symmetry with the CIBW_ENVIRONMENT syntax, which already uses a sh/bash syntax

I suppose it would need to be bundleable or pip-installable. Or we could download it at runtime like Python, but we'd need it to be from a trusted source.

Other thoughts-

• GHA already bundles a preinstalled bash. Other CId might too. We could let users choose this via an option. That would be nice, but I'd rather the defaults give the best user experience, rather than needing to opt-into it.

[JP-Ellis]

Unified Bash

Completely agree that having (nearly) the same shell across all three operating systems would be really nice.

On Windows, the simplest I have seen is the bash.exe provided as part of Git for Windows. It also comes in a portable version, which requires no installation whatsoever. This would make it very straightforward to integrate in a CI system, and can be downloaded from their GitHub release page.

There are alternatives through MSYS2, Cygwin and MinGW, and if the system is configured already, bash.exe would also be available through WSL. For reference, the GitHub Windows 2019 runner bundles three bash.exe:

Name	Target
gitbash.exe	C:\Program Files\Git\bin\bash.exe
msys2bash.cmd	C:\msys64\usr\bin\bash.exe
wslbash.exe	C:\Windows\System32\bash.exe

Alternative

One alternative I alluded to in #1779 is the possibility of having support for Python scripts directly, given that Python is obviously available. I don't know exactly how support would be added without conflicting with bash. It's just a bit of a quality of life, especially if the Python string has quotes:

• Support for Python scripts directly:

  before-test = 'import shutil; shutil.rm("{package}/bin")'
  

• Current way of doing things:

  before-build = "python -c 'import shutil; shutil.rm(\"{package}/bin\")"
  

[henryiii]

I don't think this would be a good idea. It would both break existing workflows, be very surprising on Windows to have bash syntax, and likely would cause a lot more issues like with finding MSVC on Windows. We did have this for a while on the GitHub Action, and had to revert it because some tools (like Meson) used GCC instead of MSVC from the bash shell. Note that GitHub Actions chose to select platform specific shells, and a lot of the time, you can actually get by with just writing cross platform commands. Pre-building stuff is very commonly something you'd do in before-all, and calling meson or CMake is easy to do in a way that doesn't depend on your shell, but using bash may cause those to find the wrong compilers on Windows!

You can run cibuildwheel yourself today in a bash shell on Windows by using:

- run: pipx run cibuildwheel
  shell: bash

However, this will tend to break some things, for example, if you use meson-python.

It's pretty rare to have the same commands on all platforms, and when you do, they tend to be simple. Usually you are doing things like installing packages, repairing wheels, etc. All shells support &&, all shells support running programs with args, so you can do python my_action.py or bash my_action.sh on all shells. There are a few things that would be nice to be unified (rm/del being one of the main ones), but that's very limited.

And in general, you should try to set up packages so you don't need these commands, and you shouldn't be constructing long complex commands in your pyproject.toml in a string anyway. It's much better to put the commands (if you have to have them) in an extra file and run that.

Also, finally, I think you can use triple quoted TOML strings to reduce escaping.

[henryiii]

Also, if bash was pip installable, it would probably be built by cibuildwheel, and we'd cause chicken-egg problems for every new platform that comes out.

I'm strongly in favor of the simple and expected solution of doing the native thing on every platform. It's easy for a user to use bash if that's what they want.

[skirpichev]

using bash may cause those to find the wrong compilers on Windows!

Or the right compiler. Some projects build wheels with gcc.

You can run cibuildwheel yourself today in a bash shell on Windows by using:

Indeed. Maybe the cibuildwheel action could have an option to override default shell?

[joerick]

The MSVC thing is a good point. I don't really understand the issue though, why would a subshell not inherit the required env vars from the parent?

You can run cibuildwheel yourself today in a bash shell on Windows by using:

- run: pipx run cibuildwheel
  shell: bash

It's easy for a user to use bash if that's what they want.

I'm not sure I follow this logic. Invoking cibuildwheel with bash doesn't mean that cibuildwheel uses bash for the subshells, does it? I think there might be a mechanism involving %ComSpec% to change the default windows shell, but I don't think you can put bash there.