ubuntu-latest runners have an incompatible combination of clang and libstdc++

[wheremyfoodat]

Description

Some ubuntu-latest runners come with a newer version of libstdc++ and still have clang 14 which only has partial consteval support. As a result, trying to compile a file that does #include <chrono> fails like this:

In file included from /home/runner/work/Panda3DS/Panda3DS/src/config.cpp:8:
In file included from /home/runner/work/Panda3DS/Panda3DS/third_party/toml11/toml.hpp:32:
In file included from /home/runner/work/Panda3DS/Panda3DS/third_party/toml11/toml/parser.hpp:9:
In file included from /home/runner/work/Panda3DS/Panda3DS/third_party/toml11/toml/combinator.hpp:16:
In file included from /home/runner/work/Panda3DS/Panda3DS/third_party/toml11/toml/result.hpp:5:
In file included from /home/runner/work/Panda3DS/Panda3DS/third_party/toml11/toml/traits.hpp:10:
/usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/chrono:2320:48: error: call to consteval function 'std::chrono::hh_mm_ss::_S_fractional_width' is not a constant expression
        static constexpr unsigned fractional_width = {_S_fractional_width()};
                                                      ^
/usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/chrono:2320:48: note: undefined function '_S_fractional_width' cannot be used in a constant expression
/usr/bin/../lib/gcc/x86_64-linux-gnu/13/../../../../include/c++/13/chrono:2275:2: note: declared here
        _S_fractional_width()
        ^
1 error generated.
gmake[2]: *** [CMakeFiles/Alber.dir/build.make:104: CMakeFiles/Alber.dir/src/config.cpp.o] Error 1
gmake[1]: *** [CMakeFiles/Makefile2:588: CMakeFiles/Alber.dir/all] Error 2
gmake: *** [Makefile:136: all] Error 2

Should be fixable by using clang 17.

Platforms affected

• [ ] Azure DevOps
• [X] GitHub Actions - Standard Runners
• [ ] GitHub Actions - Larger Runners

Runner images affected

• [ ] Ubuntu 20.04
• [X] Ubuntu 22.04
• [ ] macOS 11
• [ ] macOS 12
• [ ] macOS 13
• [ ] Windows Server 2019
• [ ] Windows Server 2022

Image version and build link

2.310.2

Link to a failing build: https://github.com/wheremyfoodat/Panda3DS/actions/runs/6640518979/job/18041094730 Ignore the "Setting up newer Clang" step in this run, the compilation step was still using the stock ubuntu-latest clang

Is it regression?

Yes (worked fine until yesterday: https://github.com/wheremyfoodat/Panda3DS/actions/runs/6614729350/job/17965319784)

Expected behavior

Runner clang should probably be able to compile this

Actual behavior

Some (not all) ubuntu-latest runners fail to compile C++ files that do #include <chrono>

Repro steps

1. Have any C++ code that does #include <chrono>, such as the Panda3DS repo above (without updating the compiler like it does now)
2. Set up some ubuntu-latest actions for it.
3. Specific runners should fail with the error message listed in the "Description" field

[saran-t]

Also being hit by this: https://github.com/google-deepmind/mujoco_mpc/actions/runs/6644237828/job/18052982016

I think this is only affecting C++20 builds.

[wheremyfoodat]

Oh yeah, it does only affect C++20 builds, the STL headers shouldn't touch consteval and friends if building for an earlier standard version.

[Ozaq]

Same issue for us :/ https://github.com/PedestrianDynamics/jupedsim/actions/runs/6645042045/job/18055479382

I noticed that after the image upgrade clang-15 is available but not the default clang, is this related?

Also gradual rollout fine and dandy but how am I supposed to select a newly added clang version if it is random if I get an old or new image? Should I just apt install to ensure that there is a clang-15? And why are you doing the same for the ubuntu-22.04 image? Now I have no save version to fall back on...

[wheremyfoodat]

Same issue for us :/ https://github.com/PedestrianDynamics/jupedsim/actions/runs/6645042045/job/18055479382

I noticed that after the image upgrade clang-15 is available but not the default clang, is this related?

Also gradual rollout fine and dandy but how am I supposed to select a newly added clang version if it is random if I get an old or new image? Should I just apt install to ensure that there is a clang-15? And why are you doing the same for the ubuntu-22.04 image? Now I have no save version to fall back on...

You can use the LLVM Ubuntu repository as a workaround until this is fixed. You can make it install clang-17 like here: https://github.com/wheremyfoodat/Panda3DS/blob/master/.github/workflows/Linux_Build.yml

Note that the clang command will still invoke the default clang 14, you need to use clang-17 everywhere as well.

[ilia-shipitsin]

thank you for the report. we'll have a look

[vkottler]

Maybe an anecdote, but one solution I tried (that didn't work, but seems like it should) was:

Having clang-15 / clang++-15 use --gcc-toolchain=$(dirname $(gcc-12 -print-libgcc-file-name)), but even after that + adding a number of extra hard-coded -I / -L arguments, I couldn't get figure out /usr/bin/ld: cannot find crtbeginS.o: No such file or directory (during final linking, when all other includes and library arguments were located fine and crtbeginS.o and other startup objects are in the "toolchain" directory).

Setting LD_LIBRARY_PATH to also point at the toolchain directory didn't seem to do anything (nor LIBRARY_PATH).

It seems that --gcc-toolchain= doesn't work out of the box with these Debian toolchain builds or something.

Anyway, since GCC 12 + 13 and clang 15 can currently be installed, pairing GCC 12 with clang 15 seems like an escape hatch but I wasn't personally able to get it working.

edit: this comment's suggestion to use clang 17 seems promising as an alternative

[jviotti]

Same issue here on C++20: https://github.com/sourcemeta/jsontoolkit/actions/runs/6646548576/job/18060229938. Interestingly enough, the problem is not always reproducible. Retrying the same job a couple times makes it go away.

[wheremyfoodat]

I'm not sure clang 15 can do much to fix this, Clang 17 is the one rated as having full consteval support

[ilia-shipitsin]

I'm not sure clang 15 can do much to fix this, Clang 17 is the one rated as having full consteval support

questions from our side

1. can clang 17 be installed from some kind of official repo ? (ubuntu preferrable)
2. are you already doing that in your own workflows ?
3. can/should we switch to clang 17 ?

[wheremyfoodat]

I'm not sure clang 15 can do much to fix this, Clang 17 is the one rated as having full consteval support

questions from our side

1. can clang 17 be installed from some kind of official repo ? (ubuntu preferrable)
2. are you already doing that in your own workflows ?
3. can/should we switch to clang 17 ?

1. Yes, from the official LLVM Ubuntu repository covering Ubuntu versions as early as 18.04. You can find a workflow using this and other details here: https://github.com/actions/runner-images/issues/8659#issuecomment-1780570741
2. Yep, see previously mentioned workflow (among others)
3. That's very subjective but I'd say yes honestly. It's a decently recent version of Clang with good C++20 support while being stable and tested instead of being bleeding edge. 14 is 4 majors behind by now.

[ilia-shipitsin]

I'm not sure clang 15 can do much to fix this, Clang 17 is the one rated as having full consteval support

questions from our side

1. can clang 17 be installed from some kind of official repo ? (ubuntu preferrable)
2. are you already doing that in your own workflows ?
3. can/should we switch to clang 17 ?

1. Yes, from the official LLVM Ubuntu repository covering Ubuntu versions as early as 18.04. You can find a workflow using this and other details here: ubuntu-latest runners have an incompatible combination of clang and libstdc++ #8659 (comment)
2. Yep, see previously mentioned workflow (among others)
3. That's very subjective but I'd say yes honestly. It's a decently recent version of Clang with good C++20 support while being stable and tested instead of being bleeding edge. 14 is 4 majors behind by now.

we definitely need someone with deep understanding of clang/llvm ecosystem to make a proper decision.

I've a question whether clang/llvm offers docker images (or maybe docker images from recognizable maintainer) ?

for pet project I use Fedora Rawhide image (to pick the latest clang) https://github.com/haproxy/haproxy/blob/master/.github/workflows/fedora-rawhide.yml#L14-L18

it would be really nice if clang would have been shipped like gcc: https://hub.docker.com/_/gcc

do you know are there similar images for clang/llvm ?

[wheremyfoodat]

I'm not sure clang 15 can do much to fix this, Clang 17 is the one rated as having full consteval support

questions from our side

1. can clang 17 be installed from some kind of official repo ? (ubuntu preferrable)
2. are you already doing that in your own workflows ?
3. can/should we switch to clang 17 ?

1. Yes, from the official LLVM Ubuntu repository covering Ubuntu versions as early as 18.04. You can find a workflow using this and other details here: ubuntu-latest runners have an incompatible combination of clang and libstdc++ #8659 (comment)
2. Yep, see previously mentioned workflow (among others)
3. That's very subjective but I'd say yes honestly. It's a decently recent version of Clang with good C++20 support while being stable and tested instead of being bleeding edge. 14 is 4 majors behind by now.

we definitely need someone with deep understanding of clang/llvm ecosystem to make a proper decision.

I've a question whether clang/llvm offers docker images (or maybe docker images from recognizable maintainer) ?

for pet project I use Fedora Rawhide image (to pick the latest clang) https://github.com/haproxy/haproxy/blob/master/.github/workflows/fedora-rawhide.yml#L14-L18

it would be really nice if clang would have been shipped like gcc: https://hub.docker.com/_/gcc

do you know are there similar images for clang/llvm ?

To my knowledge there's no official docker images, there's apt.llvm.org intended for providing various llvm versions via apt and then there's unofficial docker images like https://hub.docker.com/r/silkeh/clang

I have not used the latter so I can't vouch for it.

[ilia-shipitsin]

looks pretty popular

[randombit]

can/should we switch to clang 17 ?

For my usage that isn't helpful since one of the reasons we CI with Clang 14 is because that is the minimum Clang version that we support - largely because it is the default version of Clang included in 22.04. If we aren't actively testing with our minimum version we're very likely to introduce regressions, where something works with whatever Clang is in CI but not older versions.

[stenzek]

As far as I could tell, GCC 13/libstdc++ 13 getting pulled in was the result of the https://launchpad.net/~ubuntu-toolchain-r/+archive/ubuntu/test PPA getting added to the runner image. I don't know the sequence that the images are built in, but I'd guess that after the PPA was added, a general upgrade was performed, or installing something else resulted in an upgrade to gcc/libstdc++ and friends.

We've worked around this by removing the PPA, and reinstalling libstdc++, since we build AppImages from the workflows, and those were no longer compatible with Ubuntu 22.04. I'm not sure on the implications of doing so, though - perhaps the newer version was added for good reason, so if you want to do this, do so at your own risk. But it works well enough for us - see the linked PR above.

[andrea-cassioli-maersk]

Got the same issue here.

[ruifig]

As a temporary fix, I've done with oblivioncth , with the difference it checks if it's linux:

    - name: WORKAROUND FOR https://github.com/actions/runner-images/issues/8659
      if: |
        matrix.os=='ubuntu-latest'
      run: |
        echo "TEMPORARY WORKAROUND FOR GITHUB RUNNER BUG #8659\n\nRemoving GCC 13 as it breaks Clang14"
        sudo rm -f /etc/apt/sources.list.d/ubuntu-toolchain-r-ubuntu-test-jammy.list
        sudo apt-get update
        sudo apt-get install -y --allow-downgrades libc6=2.35-0ubuntu3.4 libc6-dev=2.35-0ubuntu3.4 libstdc++6=12.3.0-1ubuntu1~22.04 libgcc-s1=12.3.0-1ubuntu1~22.04

[kiwixz]

Maybe an anecdote, but one solution I tried (that didn't work, but seems like it should) was:

Having clang-15 / clang++-15 use --gcc-toolchain=$(dirname $(gcc-12 -print-libgcc-file-name)), but even after that + adding a number of extra hard-coded -I / -L arguments, I couldn't get figure out /usr/bin/ld: cannot find crtbeginS.o: No such file or directory (during final linking, when all other includes and library arguments were located fine and crtbeginS.o and other startup objects are in the "toolchain" directory).

Setting LD_LIBRARY_PATH to also point at the toolchain directory didn't seem to do anything (nor LIBRARY_PATH).

It seems that --gcc-toolchain= doesn't work out of the box with these Debian toolchain builds or something.

The correct flag is --gcc-install-dir=. It works for me with it.

[michael-projectx]

I'm also seeing this issue currently on two of my setups:

1. Ubuntu-latest using gcc 13 (c++20)
2. Ubuntu-22.04 using clang 14 (c++20)

[jviotti]

Any updates from GitHub on fixing this upstream? Otherwise we are all workarounding the same issue in a non-trivial amount of repositories. Breaking Clang C++20 builds should be treated as a base image incident.

[Adnn]

Breaking Clang C++20 builds should be treated as a base image incident.

By breaking a lot of Cpp builds by surprise for several weeks now, while not providing any convenient way to explicitly lock to a known working image version, this is hurting the case for CI.

[prenaux]

Same problem here.

[eladm-ultrawis]

also broken when using clang-16. I can't compile my C++ code. we should either have gcc version 12 or clang version 17.

[prenaux]

Its probably not a solution for everyone but we switched to the ubuntu20 image to fix the problem on our end...

[quirkylemon]

how do you switch to ubuntu20?

[oblivioncth]

how do you switch to ubuntu20?

runs-on: ubuntu-20.04

It should be noted though, for most this isn't an acceptable workaround as they need their workflows to run on both.

[eladm-ultrawis]

Is there a way to downgrade gcc 13 to gcc 12? or to remove the gcc 13 installation?

[Ozaq]

Hey GitHub Folks how come you don't rollback this change? Its ok for mistakes to happen but at this point you should probably roll back and test this internally first...

[SergiusTheBest]

I'm using https://github.com/mjp41/workaround8649 to temporary workaround the issue. You need to add the following step:

    # workaround for broken clang on ubuntu runner until https://github.com/actions/runner-images/issues/8659 get fixed
    - uses: mjp41/workaround8649@7929373c0fe5caf844d8115adccef39e3b5362e7
      with:
        os: ${{ matrix.os }}

[eladm-ultrawis]

@SergiusTheBest This is great thank you.