Add Python 3.11 support for runners

[wbarnha]

Tool name

Python

Tool license

PSF-2.0

Add or update?

• [X] Add
• [ ] Update

Desired version

3.11

Approximate size

No response

Brief description of tool

Python 3.11 was released today on https://www.python.org/downloads/, can we get this added so that it's available in python-versions and setup-python?

URL for tool's homepage

No response

Provide a basic test case to validate the tool's functionality.

No response

Platforms where you need the tool

• [ ] Azure DevOps
• [X] GitHub Actions

Runner images where you need the tool

• [X] Ubuntu 18.04
• [X] Ubuntu 20.04
• [X] Ubuntu 22.04
• [ ] macOS 10.15
• [ ] macOS 11
• [ ] macOS 12
• [ ] Windows Server 2019
• [ ] Windows Server 2022

Can this tool be installed during the build?

No response

Tool installation time in runtime

No response

Are you willing to submit a PR?

If I know where to make the necessary changes in the repo, I'd be happy to make a PR.

[erik-bershel]

@wbarnha Hi! We'll take a look.

[mikhailkoliada]

We will deploy it as soon as it is in the python-version repo

[AntonOfTheWoods]

@mikhailkoliada , is there are more detailed description of the workflow somewhere? python-version says:

Adding new versions
We are trying to build and release new versions of Python as soon as they are released. Please open an issue in [actions/virtual-environments](https://github.com/actions/virtual-environments) if any versions are missing.

Is that basically saying that the maintainers of that repo look at these issues and then add what is required there, which will then get picked up here?

[mikhailkoliada]

@AntonOfTheWoods we have a number of workflow in the python-versions repo which are responsible for repackaging a specific python version or building it from the source code and publish a new release there, then versions-manifest.json is updated with the new versions.

Finally we parse versions-manifest.json to get all the latest python major-minor versions for a specific release.

[mdshw5]

As this issue is tagged with OS: macOS, but macOS runner images were not specified in the description, will python 3.11.0 macOS builds be produced? I'm testing against 3.11.0 using the macOS image in https://github.com/mdshw5/pyfaidx/pull/197. Thanks for all your work maintaining these important resources!

[erik-bershel]

@mdshw5 similar changes are planned for macOS too.

[mgedmin]

Looks like Python 3.11 is already available for GitHub Actions. (I subscribed to this bug hoping to get a notification when that happens.)

[hugovk]

Looks like Python 3.11 is already available for GitHub Actions.

Except for macOS: https://github.com/actions/setup-python/issues/531

[mikhailkoliada]

3.11 must be available via actions/setup-python across all images, will be deployed as part of the runners next week.

[hugovk]

Looks like we're all set for 3.11 in Azure Pipelines for Linux, macOS and Windows, at least on-flight:

https://dev.azure.com/hugovk/hugovk/_build/results?buildId=2858&view=results

[jeeb]

Seems like this or some other recent update has caused something similar to #2322 to happen again, except with python3.11's 2to3-3.11. We are installing the meson build tool via homebrew, and homebrew decides it needs to update to Python 3.11, and then loudly fails when it cannot symlink /usr/local/bin/2to3-3.11.

==> Installing meson dependency: [email protected]
==> Pouring [email protected]_sur.bottle.tar.gz
Error: The `brew link` step did not complete successfully
The formula built, but is not symlinked into /usr/local
Could not symlink bin/2to3-3.11
Target /usr/local/bin/2to3-3.11
already exists. You may want to remove it:
  rm '/usr/local/bin/2to3-3.11'

To force the link and overwrite all conflicting files:
  brew link --overwrite [email protected]

To list all files that would be deleted:
  brew link --overwrite --dry-run [email protected]

Possible conflicting files are:
/usr/local/bin/2to3-3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/2to3-3.11
/usr/local/bin/idle3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/idle3.11
/usr/local/bin/pydoc3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/pydoc3.11
/usr/local/bin/python3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11
/usr/local/bin/python3.11-config -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11-config
==> /usr/local/Cellar/[email protected]/3.11.0/bin/python3.11 -m ensurepip
==> /usr/local/Cellar/[email protected]/3.11.0/bin/python3.11 -m pip install -v --no-d
==> Summary
🍺  /usr/local/Cellar/[email protected]/3.11.0: 3,163 files, 61.4MB

Can anyone else confirm?

[eli-schwartz]

I've already seen this problem for the past week in Meson's own CI, but for homebrew python3 -> [email protected], so my attempted solution was to --overwrite after brew upgrade && brew install python3. The issue is that we really, really, really need python to be homebrew python, nothing else.

As it happens to turn out, python3 from homebrew is already installed in the base image -- brew just skips it after observing it's already there -- but as described in https://github.com/Homebrew/discussions/discussions/3895 it appears that runner-images is installing homebrew python then scribbling all over its symlinks by another python.

This is unquestionably wrong, the same file is being managed and fought over by multiple sources.

It's also probably not what users actually expect, particularly if they did... NOT use actions/setup-python.

[eli-schwartz]

We're now hitting immediate brew install failures in Meson CI too, because homebrew LLVM has been rebuilt to depend on [email protected].

[szanni]

We're now hitting immediate brew install failures in Meson CI too, because homebrew LLVM has been rebuilt to depend on [email protected].

Confirmed.

 ==> Pouring [email protected]_sur.bottle.tar.gz
The formula built, but is not symlinked into /usr/local
Error: The `brew link` step did not complete successfully
Could not symlink bin/2to3-3.11
Target /usr/local/bin/2to3-3.11
already exists. You may want to remove it:
  rm '/usr/local/bin/2to3-3.11'

To force the link and overwrite all conflicting files:
  brew link --overwrite [email protected]

To list all files that would be deleted:
  brew link --overwrite --dry-run [email protected]

Possible conflicting files are:
/usr/local/bin/2to3-3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/2to3-3.11
/usr/local/bin/idle3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/idle3.11
/usr/local/bin/pydoc3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/pydoc3.11
/usr/local/bin/python3.11 -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11
/usr/local/bin/python3.11-config -> /Library/Frameworks/Python.framework/Versions/3.11/bin/python3.11-config

build.yml was working perfectly fine just a while ago.

[banach-space]

Can anyone else confirm?

Same issue in llvm-tutor when trying to install lit.

[mikhailkoliada]

Hi all, sorry for delays! We will take a closer look to conflict

[mikhailkoliada]

Deployed, though in brew python is still not updated to 3.11 on images.