runner-images icon indicating copy to clipboard operation
runner-images copied to clipboard

Ubuntu-latest workflows will use Ubuntu-24.04 image

Open ijunaidm opened this issue 1 year ago • 116 comments

Rollout will begin on December 5th and will complete on January 17th, 2025.

Breaking changes Ubuntu 24.04 is ready to be the default version for the "ubuntu-latest" label in GitHub Actions and Azure DevOps.

Target date This change will be rolled out over a period of several weeks beginning December 5th and will complete on January 17th, 2025.

The motivation for the changes GitHub Actions and Azure DevOps have supported Ubuntu 24.04 in preview mode since May 2024, and starting from July 2024 Ubuntu 24.04 is generally available for all customers. We have monitored customer feedback to improve the Ubuntu 24.04 image stability and now we are ready to set it as the latest. There are a set of packages listed below that we have removed from the Ubuntu 24 image. Please review the list carefully to see if you will be impacted by these changes. We have made cuts to the list of packages so that we can maintain our SLA for free disk space. The images have grown so large we are in danger of violating our SLA if we keep the package list as-is.

The factors we took into consideration when removing packages are as follows:

  • How long does it take to install the tool at runtime?
  • How much space does it take up on the image?
  • How many users are there of the tool?

We understand that our reasoning may not make sense to some of you out there, but please bear in mind that we tried to keep disruptions as minimal as possible, and tried to keep the best interests of the community at large in mind. There is a very large and diverse community using our images, and as much as we would like to, we cannot pre-install every tool on these images.

Platforms affected

  • [x] Azure DevOps
  • [x] GitHub Actions

Mitigation ways Steps or options for impact mitigation If you see any issues with your workflows during transition period:

  • Switch back to Ubuntu 22 by changing workflow YAML to use runs-on: ubuntu-22.04 We support two latest LTS Ubuntu versions, so Ubuntu 22 will still be maintained for the next 2 years.
  • File an issue in this repository

Software Differences The Ubuntu 22.04 image has a different set of software than Ubuntu 24.04. The most significant changes are listed in the table below:

Tool name Ubuntu 22.04 Ubuntu 24.04 Notes
Clang
  • 13.*
  • 14.* (default)
  • 15.*
  • 16.*
  • 17.*
  • 18.* (default)
The most recent versions are installed
GCC / GNU C++ / GNU Fortran
  • 9.*
  • 10.*
  • 11.*
  • 12.*
  • 13.*
  • 12.*
  • 13.*
  • 14.*
The most recent versions are installed
PHP 8.1.* 8.3.* The most recent version are installed
Java
  • 8.*
  • 11.* (default)
  • 17.*
  • 21.*
  • 8.*
  • 11.*
  • 17.* (default)
  • 21.*
Default Java switched to 17.* for Ubuntu 24.04 image.
Python
  • 3.7.* (cached)
  • 3.8.* (cached)
  • 3.9.* (cached)
  • 3.10.* (default)
  • 3.11.* (cached)
  • 3.12.* (cached)
  • 3.9.* (cached)
  • 3.10.* (cached)
  • 3.11.* (cached)
  • 3.12.* (default)
Pre-cached versions currently unavailable. Default version switched to the latest one. On GitHub Actions, actions/setup-python can install any version on-flight so this change doesn't impact users
Go
  • 1.20.* (cached)
  • 1.21.* (default)
  • 1.22.* (cached)
  • 1.21.* (cached)
  • 1.22.* (cached)
  • 1.23.* (default)
If your use-case requires using any of these versions, consider using tasks to install Go on-flight:
PyPy
  • 3.7.* (cached)
  • 3.8.* (cached)
  • 3.9.* (cached)
  • 3.10.* (cached)
  • 3.9.* (cached)
  • 3.10.* (cached)
Deprecated all the versions besides the most recent ones
Ruby
  • 3.0.* (default)
  • 3.1.* (cached)
  • 3.2.* (default)
We need to update 3.3.5
Node.js
  • 16.* (cached)
  • 18.* (default)
  • 20.* (cached)
  • 16.* (cached)
  • 18.* (cached)
  • 20.* (default)
If your use-case requires using any of these versions, consider using tasks to install Node.js on-flight:
Heroku latest available - Removed from the Ubuntu 24.04 image due to maintenance reasons.
Leiningen latest available - Removed from the Ubuntu 24.04 image due to maintenance reasons.
Mono / MSBuild / NuGet latest available - Software is not available for Ubuntu 24 at the moment.
Terraform latest available - Removed from the Ubuntu 24.04 image due to maintenance reasons.
R latest available - Removed from the Ubuntu 24.04 image due to maintenance reasons.
SVN latest available - Removed from the Ubuntu 24.04 image due to maintenance reasons.
Alibaba Cloud CLI latest available - Removed from the Ubuntu 24.04 image due to maintenance reasons.
Netlify CLI latest available - Removed from the Ubuntu 24.04 image due to maintenance reasons.
OpenShift CLI latest available - Removed from the Ubuntu 24.04 image due to maintenance reasons.
ORAS CLI latest available - Removed from the Ubuntu 24.04 image due to maintenance reasons.
Vercel CLI latest available - Removed from the Ubuntu 24.04 image due to maintenance reasons.
Bindgen / Cbindgen latest available - Removed from the Ubuntu 24.04 image due to maintenance reasons.
Cargo audit/clippy/outdated latest available - Removed from the Ubuntu 24.04 image due to maintenance reasons.
.NET Core SDK
  • 6.*
  • 7.*
  • 8.*
  • 8.*
Please consider using tasks to install any version on-flight:
PostgreSQL 14.* 16.* More recent version are installed
MS SQL Server Client Tools sqlcmd / SqlPackage - Removed from the Ubuntu 24.04 image due to maintenance reasons.
MarkdownPS Module latest available - Removed from the Ubuntu 24.04 image due to maintenance reasons.
Android Command Line Tools 9.0 12.0 The most recent version are installed
Android SDK Build-tools
  • 34.0.0
  • 33.0.0
  • 33.0.1
  • 33.0.2
  • 33.0.3
  • 32.0.0
  • 31.0.0
  • 34.0.0
The most recent version are installed
Android NDK
  • 25.* (default)
  • 26.*
  • 27.* (default)
  • 26.*
The most recent version are installed
Cached Docker images
  • alpine:3.16
  • alpine:3.17
  • alpine:3.18
  • debian:10
  • debian:11
  • moby/buildkit:latest
  • node:16
  • node:16-alpine
  • node:18
  • node:18-alpine
  • node:20
  • node:20-alpine
  • ubuntu:20.04
  • ubuntu:22.04
- Removed from the Ubuntu 24.04 image due to maintenance reasons.

ijunaidm avatar Sep 17 '24 18:09 ijunaidm

Nuget and sqlpackage not being available is a major showstopper for us.

The more I look at this list the more I realize that 24.04 is a step back for a lot of people. People are going to complain when this becomes the ubuntu-latest. Be aware of this and maybe make sure this communication is shown on github/azure devops

But thanks for the heads up!

ThibaultLesuisse avatar Sep 17 '24 18:09 ThibaultLesuisse

@ijunaidm

ubuntu-24.04 is still labeled beta in the overview on https://github.com/actions/runner-images/blob/main/README.md#available-images

image

Will you now be removing the beta tag?

MikeMcC399 avatar Sep 19 '24 07:09 MikeMcC399

Yes @MikeMcC399 . Its updated and removed.

ijunaidm avatar Sep 19 '24 20:09 ijunaidm

@ThibaultLesuisse are you aware that NuGet refers specifically to NuGet.exe (which requires mono on Mac & Linux), as does Azure Pipeline's NuGetCommand, and GitHub Actions' nuget/setup-nuget action?

If you're building your projects with dotnet build, you're much better off restoring with dotnet restore, and not using NuGet.exe or mono at all.

zivkan avatar Sep 25 '24 01:09 zivkan

But we need NuGet.exe to sync our custom Artifacts store. And yes I know that commonly you need to run mono nuget.exe to run it on Mac/Linux. But it doesn't change the fact that a lot of tools are now missing for various reasons. This should be communicated.

ThibaultLesuisse avatar Sep 25 '24 08:09 ThibaultLesuisse

We're also affected by the removal of Mono.

The official Mono repo only lists 20.04, but the package seem to work fine in newer versions–which is how the 22.04 image got Mono:

https://github.com/actions/runner-images/blob/ae99c16b0cf27c0cca7ef0b08e7bcf13c33f6cfa/images/ubuntu/scripts/build/install-mono.sh#L15-L18

Is there a reason this can't be done for 24.04 as well? I tested it briefly and it's working for our relatively simple needs.

Mono is fairly chunky so we'd rather not install it every single workflow run.

PathogenDavid avatar Sep 25 '24 19:09 PathogenDavid

Hi there - my team is scrambling this morning due to this change. I have detailed the issue on the community forum here, but the tl;dr is that this version bump prevents Python packages from being installed for system Python. Unfortunately, it seems the evaluation of this issue was incorrect:

On GitHub Actions, actions/setup-python can install any version on-flight so this change doesn't impact users

In fact, my team has a large number of workflows created by many different engineers, and around a dozen of our workflows that didn't previously leverage actions/setup-python now have to go through our internal change management process in order to restore stability to our CI/CD

In the future, please announce breaking changes as a deprecation notice on the GitHub Blog. It's the one place we rely on to proactively catch and prevent stability issues from occurring in our pipelines

sayhiben avatar Sep 26 '24 17:09 sayhiben

This morning, several of our CI pipelines broke due to these changes. It would be great to have compatibility with the following dependencies, allowing us to upgrade the ubuntu version in the future without complicating our pipelines :bow: .

  • `gcloud
  • `sbt
  • libncurses.so.5

obluff avatar Sep 26 '24 20:09 obluff

We have had issues today due to the change. We are getting this error message when using ubuntu-latest

 unable to load shared object '/home/runner/work/_temp/Library/ragg/libs/ragg.so':
  libtiff.so.5: cannot open shared object file: No such file or directory

When used the previous version ubuntu-22.04 everything runs without issue. I am unsure of what I need to update in our Action scripts to fix this issue.

aylapear avatar Sep 30 '24 20:09 aylapear

I think I'm experiencing the same problem as @sayhiben - without either adding an extra step to a Ruby workflow to install an older Python version or deliberately requesting ubuntu-22.04 to avoid the problem completely, any GitHub Package installer that uses pip to install system-level packages inside the container is going to fail with the following output:

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
    sure you have python3-full installed.
    
    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you. Make sure you have pipx installed.
    
    See /usr/share/doc/python3.12/README.venv for more information.

oc-liz-conlan avatar Oct 01 '24 14:10 oc-liz-conlan

-Readme.md

At252 avatar Oct 02 '24 00:10 At252

I am trying to use arm-runner-action with the base specified at ubuntu-latest. Sometime in the last two weeks my workflow for arm platform started failing because qemu was not getting configured/built?/whatever correctly for the new ubuntu-latest. The work around is to specify ubuntu-22.04. The error using ubuntu-latest with [email protected] is:

...
Setting up g++ (4:10.2.1-1) ...
update-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode
Setting up dpkg-dev (1.20.13) ...
Setting up build-essential (12.9) ...
Processing triggers for libc-bin (2.31-13+rpt2+rpi1+deb11u10) ...
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault (core dumped)
qemu: uncaught target signal 11 (Segmentation fault) - core dumped
Segmentation fault (core dumped)
dpkg: error processing package libc-bin (--configure):
 installed libc-bin package post-installation script subprocess returned error exit status 139
Errors were encountered while processing:
 libc-bin
E: Sub-process /usr/bin/dpkg returned an error code (1)
Error: Process completed with exit code 100.

mcmillan03 avatar Oct 07 '24 19:10 mcmillan03

Removing gcloud is going to break a bunch of our pipelines, this is a pretty major breaking set of changes to remove all these packages.

lukeschlather avatar Oct 07 '24 20:10 lukeschlather

Would you please kindly update the README and put the ubuntu-latest YAML label back to 22.04 until the migration is fully over and this PR is closed? We found this the hard way in our workflow when we assumed a version of one of the tools only to realize the underlying host was something else than advertised.

eskultety avatar Oct 09 '24 08:10 eskultety

to add something positive: looking forward to an updated curl version by default

Loki-Afro avatar Oct 09 '24 08:10 Loki-Afro

Our pipelines broke this morning because Terraform is no longer included. While I have no issue with it being removed (these are GitHub's images and I'm consciously risking my pipelines breaking by relying on ubuntu-latest instead of pinning) it would be nice to have a little more explanation as to why TF was removed (and tools like bicep and packer left in). It's now just in a list of other removed tools with the remark "Maintenance reasons". Can anyone elaborate?

mdirkse avatar Oct 09 '24 09:10 mdirkse

Is there an actual reason as to why nuget cli was removed?

telmo-rodrigues-yld avatar Oct 09 '24 15:10 telmo-rodrigues-yld

We were suddenly hit by removed terraform too... Any particular reason for it? Is it gonna be a permanent change or will you work on bringing it back?

pkuczynski avatar Oct 10 '24 12:10 pkuczynski

brainstorm idea: would be nice to have <image type>-beta labels available for use in periodic workflows

I usually leave the images specified as <image type>-latest and accept that my workflows will maybe break without notice, because the changes are usually easy to handle and it keeps my repos current

But since the upcoming releases are just tagged by specific version numbers until the latest tag slides to them, there's no way to have a stable workflow that pre-tests things and finds the breaks before they hit our important pipelines

If there was a stable name I could do matrix expansion on for scheduled checks, it might help folks doing their best to stay current but without unexpected issues

Cheers

mikehardy avatar Oct 11 '24 14:10 mikehardy

This morning several of our CI pipelines broke as well because they access the Linux keyring. It would nice to have the libsecret-1-dev dependency restored in Ubuntu 24 since it was part of the image for previous versions.

t1m0thyj avatar Oct 14 '24 13:10 t1m0thyj

Not sure if this is now the place to ask or not but since others are, we were using liblzma I suppose with the removal of a huge huge pile of packages, all their transitives are gone as well which is why the libraries are gone. Not too hard to install them at least.

mikehardy avatar Oct 14 '24 15:10 mikehardy

~Similarly, I think that liblapack3 was lost during the upgrade from 2204 to 2404.~

Sorry, I take that back. The library was simply updated, but we used a binary that was looking for an older version of the shared library yielding the error. We worked around the issue simply by fixing the Ubuntu version for now.

rschwarz avatar Oct 14 '24 15:10 rschwarz

See #10783

kojiromike avatar Oct 15 '24 02:10 kojiromike

Referencing this issue here - #10781 is caused by this. Documentation should be updated to reflect this breaking change, or the interpreter of python that comes with the runner image should be installed independently from the interpreter that ships with ubuntu so that it is no longer externally managed.

This change WILL break a lot of actions and existing workflows which target the ubuntu-latest label.

molson504x avatar Oct 15 '24 17:10 molson504x

Hello! This update has broken my ubuntu end-to-end tests with Playwright + Electron. Need to investigate why. For now I'll pin to runs-on: ubuntu-22.04.

itsdouges avatar Oct 15 '24 20:10 itsdouges

If you're building your projects with dotnet build, you're much better off restoring with dotnet restore, and not using NuGet.exe or mono at all.

And what about nuget pack? It crashes even if I'm using dotnet restore and dotnet build image

Saibamen avatar Oct 15 '24 21:10 Saibamen

See https://github.com/actions/runner-images/issues/10788 - the new image fails to include nvm and sbt. (setup-node is not a sufficiently capable replacement for nvm, ftr)

Please reverse some of these very ill-advised breaking changes, and please consider using notification to alert any repo owners that would be affected by these kinds of breaking updates.

Especially since everyone's been suggested to use "latest" for years, it's just not acceptable to ever ship breaking changes that aren't absolutely necessary.

ljharb avatar Oct 15 '24 21:10 ljharb

This broke all of our playwright tests. I guess you have to crack a few eggs to make an omelet, but I'm missing some "official" word on what is going on. From what I understand, all ubuntu-latest now run on 24.04, and it is fundamentally broken. Is this being worked on? Pinning everything we have running on non-private runners to 22.04 for now...

Ryangr0 avatar Oct 16 '24 09:10 Ryangr0

From what I understand, all ubuntu-latest now run on 24.04, and it is fundamentally broken.

Unfortunately, that's not true. Some of the ubuntu-latest run on 24.04, others are still 22.04 even though the documentation says something else. I wouldn't say 24.04 is fundamentally broken, it just differs from 22.04 and the communication is far from ideal.

jajik avatar Oct 16 '24 09:10 jajik

From what I understand, all ubuntu-latest now run on 24.04, and it is fundamentally broken.

Unfortunately, that's not true. Some of the ubuntu-latest run on 24.04, others are still 22.04 even though the documentation says something else. I wouldn't say 24.04 is fundamentally broken, it just differs from 22.04 and the communication is far from ideal.

Yes. On master repo, ubuntu-latest runs on Ubuntu LTS 24.04.1, but on my fork it runs in 22.04.5 (see: https://github.com/NuGet/setup-nuget/issues/168).

Saibamen avatar Oct 16 '24 09:10 Saibamen