runner-images icon indicating copy to clipboard operation
runner-images copied to clipboard

Building on windows-latest sporadically has random LLVM version for bindgen

Open vlovich opened this issue 6 months ago • 8 comments

Description

When I run Rust's bindgen on windows-latest, sometimes the reported version is 18.1 and sometimes it's 19.

The bindgen tool fails with:

  Failed to generate bindings: ClangDiagnostic("C:\\Program Files\\Microsoft Visual Studio\\2022\\Enterprise\\VC\\Tools\\MSVC\\14.44.35207\\include\\yvals_core.h:908:1: error: static assertion failed: error STL1000: Unexpected compiler version, expected Clang 19.0.0 or newer.\n")

because the headers in MSVC are newer than the last supported version of clang. This started sporadically breaking recently.

Platforms affected

  • [ ] Azure DevOps
  • [x] GitHub Actions - Standard Runners
  • [ ] GitHub Actions - Larger Runners

Runner images affected

  • [ ] Ubuntu 22.04
  • [ ] Ubuntu 24.04
  • [ ] macOS 13
  • [ ] macOS 13 Arm64
  • [ ] macOS 14
  • [ ] macOS 14 Arm64
  • [ ] macOS 15
  • [ ] macOS 15 Arm64
  • [ ] Windows Server 2019
  • [x] Windows Server 2022
  • [ ] Windows Server 2025

Image version and build link

No public example

Is it regression?

Yes. Works: https://github.com/actions/runner-images/blob/win22/20250609.2/images/windows/Windows2022-Readme.md Broken: https://github.com/actions/runner-images/blob/win22/20250617.1/images/windows/Windows2022-Readme.md

Expected behavior

The build capability shouldn't have regressed & bindgen should still be working.

Actual behavior

Bindgen is failing

Repro steps

windows-latest seems to be inconsistent about the image it's selecting (maybe incremental rollout?) and sometimes it's picking a broken image.

vlovich avatar Jun 20 '25 16:06 vlovich

Hi @vlovich, Thank you for bringing this issue to our attention. We will look into this issue and will update you after investigating.

hemanthmanga avatar Jun 20 '25 17:06 hemanthmanga

can reproduce outside of the context of Rust development as well - just building C++ binaries with Clang reports the exact same error (which makes sense since the error is the STL expecting a newer compiler version)

[1/11] Building CXX object _deps/fmt-build/CMakeFiles/fmt.dir/src/format.cc.obj
FAILED: [code=1] _deps/fmt-build/CMakeFiles/fmt.dir/src/format.cc.obj 
C:\Users\runneradmin\.cargo\bin\sccache.exe C:\PROGRA~1\LLVM\bin\CLANG_~1.EXE -D_HAS_ITERATOR_DEBUGGING=0 -ID:/a/Dash-Orb-Practice-Fix/Dash-Orb-Practice-Fix/cpm-cache/fmt/69912fb6b71fcb1f7e5deca191a2bb4748c4e7b6/include -O3 -DNDEBUG -std=gnu++20 -D_DLL -D_MT -Xclang --dependent-lib=msvcrt -flto=thin --target=x86_64-pc-windows-msvc -MD -MT _deps/fmt-build/CMakeFiles/fmt.dir/src/format.cc.obj -MF _deps\fmt-build\CMakeFiles\fmt.dir\src\format.cc.obj.d -o _deps/fmt-build/CMakeFiles/fmt.dir/src/format.cc.obj -c D:/a/Dash-Orb-Practice-Fix/Dash-Orb-Practice-Fix/cpm-cache/fmt/69912fb6b71fcb1f7e5deca191a2bb4748c4e7b6/src/format.cc
In file included from D:/a/Dash-Orb-Practice-Fix/Dash-Orb-Practice-Fix/cpm-cache/fmt/69912fb6b71fcb1f7e5deca191a2bb4748c4e7b6/src/format.cc:8:
In file included from D:/a/Dash-Orb-Practice-Fix/Dash-Orb-Practice-Fix/cpm-cache/fmt/69912fb6b71fcb1f7e5deca191a2bb4748c4e7b6/include\fmt/format-inl.h:12:
In file included from C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\algorithm:8:
C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\yvals_core.h:908:1: error: static assertion failed: error STL1000: Unexpected compiler version, expected Clang 19.0.0 or newer.
  908 | _EMIT_STL_ERROR(STL1000, "Unexpected compiler version, expected Clang 19.0.0 or newer.");
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\yvals_core.h:534:58: note: expanded from macro '_EMIT_STL_ERROR'
  534 | #define _EMIT_STL_ERROR(NUMBER, MESSAGE)   static_assert(false, "error " #NUMBER ": " MESSAGE)
      |                                                          ^~~~~
1 error generated.
[2/11] Building CXX object _deps/fmt-build/CMakeFiles/fmt.dir/src/os.cc.obj
FAILED: [code=1] _deps/fmt-build/CMakeFiles/fmt.dir/src/os.cc.obj 
C:\Users\runneradmin\.cargo\bin\sccache.exe C:\PROGRA~1\LLVM\bin\CLANG_~1.EXE -D_HAS_ITERATOR_DEBUGGING=0 -ID:/a/Dash-Orb-Practice-Fix/Dash-Orb-Practice-Fix/cpm-cache/fmt/69912fb6b71fcb1f7e5deca191a2bb4748c4e7b6/include -O3 -DNDEBUG -std=gnu++20 -D_DLL -D_MT -Xclang --dependent-lib=msvcrt -flto=thin --target=x86_64-pc-windows-msvc -MD -MT _deps/fmt-build/CMakeFiles/fmt.dir/src/os.cc.obj -MF _deps\fmt-build\CMakeFiles\fmt.dir\src\os.cc.obj.d -o _deps/fmt-build/CMakeFiles/fmt.dir/src/os.cc.obj -c D:/a/Dash-Orb-Practice-Fix/Dash-Orb-Practice-Fix/cpm-cache/fmt/69912fb6b71fcb1f7e5deca191a2bb4748c4e7b6/src/os.cc
In file included from D:/a/Dash-Orb-Practice-Fix/Dash-Orb-Practice-Fix/cpm-cache/fmt/69912fb6b71fcb1f7e5deca191a2bb4748c4e7b6/src/os.cc:13:
In file included from D:/a/Dash-Orb-Practice-Fix/Dash-Orb-Practice-Fix/cpm-cache/fmt/69912fb6b71fcb1f7e5deca191a2bb4748c4e7b6/include\fmt/os.h:11:
In file included from D:/a/Dash-Orb-Practice-Fix/Dash-Orb-Practice-Fix/cpm-cache/fmt/69912fb6b71fcb1f7e5deca191a2bb4748c4e7b6/include\fmt\format.h:41:
In file included from D:/a/Dash-Orb-Practice-Fix/Dash-Orb-Practice-Fix/cpm-cache/fmt/69912fb6b71fcb1f7e5deca191a2bb4748c4e7b6/include\fmt\base.h:20:
In file included from C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\type_traits:8:
C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\yvals_core.h:908:1: error: static assertion failed: error STL1000: Unexpected compiler version, expected Clang 19.0.0 or newer.
  908 | _EMIT_STL_ERROR(STL1000, "Unexpected compiler version, expected Clang 19.0.0 or newer.");
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\yvals_core.h:534:58: note: expanded from macro '_EMIT_STL_ERROR'
  534 | #define _EMIT_STL_ERROR(NUMBER, MESSAGE)   static_assert(false, "error " #NUMBER ": " MESSAGE)
      |                                                          ^~~~~
1 error generated.

Cvolton avatar Jun 20 '25 19:06 Cvolton

@Cvolton right but simultaneously it's weird that I'm observing flapping in our CI. I noted the image it seemed like it was working on vs broken which ostensibly in the release notes says it's Clang 18 yet I seem to observe Clang 19 in one and Clang 18 in the other.

vlovich avatar Jun 20 '25 22:06 vlovich

-D_ALLOW_COMPILER_AND_STL_VERSION_MISMATCH is helped for me for now, but of course version mismatch should be fixed in the image (update to Clang 19).

m6vrm avatar Jun 21 '25 11:06 m6vrm

Do you need more logs of the same error? able to reproduce on windows-latest

ZeunO8 avatar Jun 23 '25 07:06 ZeunO8

Seeing the same error on my clang-cl builds (pure C++, no Rust). E.g., see: https://github.com/JoostHouben/Euwe-chess-engine/actions/runs/15818282100/job/44581369301?pr=40#step:10:434

So far the error seems quite consistent to me, not random.

Is it possible to select the clang-cl binary that comes with Visual Studio, instead of the one from the LLVM toolchain installed on the runner image?

JoostHouben avatar Jun 23 '25 07:06 JoostHouben

Don't get this error using windows-2025

ZeunO8 avatar Jun 23 '25 15:06 ZeunO8

Please upgrade the clang version to 19 or above. Please use either of the below steps asa workaround -

 - name: Install LLVM Clang 19 on Windows
        shell: powershell
        run: |
         $ErrorActionPreference = "Stop"
         Invoke-WebRequest -Uri "https://github.com/llvm/llvm-project/releases/download/llvmorg-19.1.7/LLVM-19.1.7-win64.exe" -OutFile "llvm.exe"
         Start-Process -FilePath .\llvm.exe -ArgumentList "/S", "/D=C:\LLVM" -Wait
         echo "C:\LLVM\bin" | Out-File -FilePath $env:GITHUB_PATH -Encoding utf8 -Append

== OR ==

      - name: Set up Clang
        uses: egor-tensin/setup-clang@v1
        with:
          version: 19.1.7
          platform: x64

The visual studio version was upgraded to 17.14 which needs minimum clang version of 19.1.5

subir0071 avatar Jun 23 '25 19:06 subir0071

We are in the process of upgrading llvm to the compatible version 20 for Win22. https://github.com/actions/runner-images/pull/12501

subir0071 avatar Jun 30 '25 17:06 subir0071

We are in the process of upgrading llvm to the compatible version 20 for Win22. #12501

Are you able to use clang+llvm-20.1.7-x86_64-pc-windows-msvc.tar.xz as it contains extra tooling liblldb.lib for example, as well as CMake configs

ZeunO8 avatar Jun 30 '25 22:06 ZeunO8

Hi @vlovich - Could you please try now with latest image rollout and confirm the same .

cc @Cvolton @ZeunO8

kishorekumar-anchala avatar Jul 03 '25 08:07 kishorekumar-anchala

Hi @vlovich - Could you please try now with latest image rollout and confirm the same .

cc @Cvolton @ZeunO8

I'm getting this error all the time

cho-regin avatar Jul 03 '25 09:07 cho-regin

@kishorekumar-anchala no unfortunately i can't. We've switched to windows 2025 because our CI was blocked on this.

vlovich avatar Jul 03 '25 12:07 vlovich

Hi @vlovich , @Cvolton @cho-regin

  1. I suspect this issue is related to a Visual Studio component. I anticipate it will be resolved with the next image rollout and will share an update once that is completed.

Working versions

Microsoft.VisualStudio.Component.VC.Llvm.Clang 17.13.35710.127
Microsoft.VisualStudio.Component.VC.Llvm.ClangToolset 17.13.35710.127

failing versions

Microsoft.VisualStudio.Component.VC.Llvm.Clang 17.14.36015.10
Microsoft.VisualStudio.Component.VC.Llvm.ClangToolset 17.14.36015.10
  1. Could you please confirm if you're working on Visual Studio project that builds your LLVM-based app ?

kishorekumar-anchala avatar Jul 03 '25 13:07 kishorekumar-anchala

@kishorekumar-anchala: It is a CMake project that uses Ninja as generator. If that answers your question.

cho-regin avatar Jul 03 '25 13:07 cho-regin

https://github.com/boostorg/uuid/actions/runs/16060338375/job/45401613779

**********************************************************************
** Visual Studio 2022 Developer Command Prompt v17.14.5
** Copyright (c) 2025 Microsoft Corporation
**********************************************************************
[vcvarsall.bat] Environment initialized for: 'x64'
###
### Using 'vc143' toolset.
###
...
     "clang-cl.exe" --target=i386-pc-windows-msvc "libs\uuid\test\compile\decl_header.cpp" -c -Fo"bin.v2\libs\uuid\test\~hdr-decl-time_generator~hpp.test\clang-win-18.1.8\debug\x86_32\cxxstd-14-iso\~hdr-decl-time_generator~hpp.obj"     -TP /EHs /std:c++14 /GR /Z7 /Od /Ob0 /W4 /WX /MDd -DBOOST_ALL_NO_LIB=1 -DBOOST_UUID_TEST_HEADER=time_generator.hpp "-I." "-ID:\a\uuid\boost-root\libs\assert\include" "-ID:\a\uuid\boost-root\libs\core\include" "-ID:\a\uuid\boost-root\libs\static_assert\include" "-ID:\a\uuid\boost-root\libs\throw_exception\include" "-ID:\a\uuid\boost-root\libs\type_traits\include" "-ID:\a\uuid\boost-root\libs\uuid\include" "-Ilibs\config\include" 

...failed compile-c-c++ bin.v2\libs\uuid\test\~hdr-decl-time_generator~hpp.test\clang-win-18.1.8\debug\x86_32\cxxstd-14-iso\~hdr-decl-time_generator~hpp.obj...
compile-c-c++ bin.v2\libs\uuid\test\~hdr-decl-time_generator_v1~hpp.test\clang-win-18.1.8\debug\x86_32\cxxstd-14-iso\~hdr-decl-time_generator_v1~hpp.obj
In file included from libs\uuid\test\compile\decl_header.cpp:19:
In file included from .\boost/uuid/time_generator_v1.hpp:8:
In file included from .\boost/uuid/uuid.hpp:9:
In file included from .\boost/uuid/uuid_clock.hpp:8:
In file included from C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\chrono:8:
C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\yvals_core.h(908,1): error: static assertion failed: error STL1000: Unexpected compiler version, expected Clang 19.0.0 or newer.
  908 | _EMIT_STL_ERROR(STL1000, "Unexpected compiler version, expected Clang 19.0.0 or newer.");
      | ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Tools\MSVC\14.44.35207\include\yvals_core.h(534,58): note: expanded from macro '_EMIT_STL_ERROR'
  534 | #define _EMIT_STL_ERROR(NUMBER, MESSAGE)   static_assert(false, "error " #NUMBER ": " MESSAGE)
      |                                                          ^~~~~
1 error generated.

jeking3 avatar Jul 05 '25 13:07 jeking3

Apologies for the delay, the Clang upgrade in the image appears to be working for me - I have removed the manual forced upgrade workaround I had previously and the build system successfully identifies Clang 20.1.7 with GNU-like command-line and I currently don't seem to be getting any errors. Will keep an eye out to see if I observe any flip-flopping like the OP though (the issue was consistent for me in the past though)

My use case is a CMake project using Ninja as the generator and Clang as the compiler with the target being a Windows .dll - Clang uses the STL from Microsoft Visual Studio there

Cvolton avatar Jul 09 '25 20:07 Cvolton

Hi @vlovich Kindly verify now with latest image rollout and confirm back . Thank you !

cc @Cvolton @cho-regin

kishorekumar-anchala avatar Jul 13 '25 17:07 kishorekumar-anchala

Hi, I have only done one test run so far, but my project build appears to be working fine so far. Thank you for the fixes.

Cvolton avatar Jul 13 '25 21:07 Cvolton