[vcpkg-tool] cache fail in GHA

[talregev]

Operating system

windows github action

Compiler

msvc

Steps to reproduce the behavior

vcpkg install xxx

Failure logs

https://github.com/curl/curl/pull/17069

More detail on curl PR.

Additional context

No response

[talregev]

CC @vszakats

[vszakats]

Some more experiments here: https://github.com/curl/curl/pull/17071

Tried downgrading vcpkg-tool. Tried dropping windows-11-arm runner from the workflow.

edit: correction: I think it started while I was playing with adding windows-2025 runner a day before (Monday). I was surprised that packages are being rebuilt throughout this, with the same MSVC version and settings. Probably irrelevant.

[vszakats]

Here's a vcpkg --debug log about trying to upload to the cache, then reporting that zero upload has been done:

Starting submission of vcpkg-cmake:x64-linux@2024-04-23 to 1 binary cache(s) in the background
[...]
[DEBUG] 1025: execute_process(curl -s -L -H "User-Agent: vcpkg/2025-03-13-7699e411ea11543de6abc0c7d5fd11cfe0039ae5 (curl)" -H "Accept: application/json;api-version=6.0-preview.1" -H "Content-Type: application/json" -H "Authorization: ***" -X POST --data-raw "{
  \"key\": \"vcpkg-cmake-420a1c6ff1038356b66b2ffb7212025ab6f913d0a057a948337a6e15e85ce226\",
  \"version\": \"420a1c6ff1038356b66b2ffb7212025ab6f913d0a057a948337a6e15e85ce226\",
  \"cacheSize\": 10087
}
"
https://acghubeus2.actions.githubusercontent.com/LBtO2h2PnCINU7gm3jjmEjodUj2g1Be2AKcmGQDkydACxmcFgb/_apis/artifactcache/caches)
[...]
Completed submission of vcpkg-cmake:x64-linux@2024-04-23 to 0 binary cache(s) in 127 ms

https://github.com/curl/curl/actions/runs/14492605120/job/40652434807?pr=17071#step:5:708

This happens with all packages sent up to the cache.

Packages not showing up in the cache: https://github.com/curl/curl/actions/caches

[jdpurcell]

Could be related to https://github.blog/changelog/2025-03-20-notification-of-upcoming-breaking-changes-in-github-actions/ ?

[vszakats]

@jdpurcell Thanks for the pointer. It may be related. Our workflow does edit ACTIONS_CACHE_URL and ACTIONS_RUNTIME_TOKEN. The page suggests this means relying on an old cache API. This also explains the sudden disappearance of the cache entries.

curl's setup follows these instructions, which may be outdated in the light of the GitHub announcement: https://learn.microsoft.com/vcpkg/consume/binary-caching-github-actions-cache https://learn.microsoft.com/vcpkg/reference/binarycaching#gha https://github.com/microsoft/vcpkg-tool/blob/f9b6c6917b23c1ccf16c1a9f015ebabf8f615045/src/vcpkg/binarycaching.cpp#L1844

I wonder if it means that the vcpkg binary cache feature fell into the cracks of this deprecation and now unavailable? (The docs made it clear from the get go that this feature may disappear anytime. This would nevertheless be fatal for curl's use case.)

If still available, what action to take to keep using it?

[ADKaster]

This affects LadybirdBrowser/ladybird in a very negative way as well, bringing our CI times from 10-20 minutes to 60-100 minutes per job. An update to the documentation on how to make"VCPKG_BINARY_SOURCES": "clear;x-gha,readwrite" work again would be much appreciated, and probably take a lot of minutes off of GHA runners for projects that were relying on this feature.

[talregev]

cc @BillyONeal

[HunterZ]

I think I'm seeing the same thing: https://github.com/microsoft/vcpkg-docs/issues/473#issuecomment-2816468599

[magnesj]

We use vcpg and cache artifacts, the build servers now use 2-3 hours to build the required vcpkg binaries. https://github.com/OPM/ResInsight

Please provide guidance on how to move forward.

[rustyconover]

Seems like I'm running into this as well building duckdb extensions that use vcpkg.

From a workflow:

[DEBUG] 1161: execute_process(curl -s -L -H "User-Agent: vcpkg/2024-11-12-eb492805e92a2c14a230f5c3deb3e89f6771c321 (curl)" -H "Accept: application/json;api-version=6.0-preview.1" -H "Content-Type: application/json" -H "Authorization: ***" -X POST --data-raw "{
  \"key\": \"vcpkg-boost-306daf81a520f4add175100ac70e74920e14e12d652a5b8c6a98de19e4d988c9\",
  \"version\": \"306daf81a520f4add175100ac70e74920e14e12d652a5b8c6a98de19e4d988c9\",
  \"cacheSize\": 5921
}
" https://acghubeus1.actions.githubusercontent.com/3e7hfOg6omJcp3e2ihsBHbIEVGL8M2XqyVr0FPSVNt5PURu7GQ/_apis/artifactcache/caches)
[DEBUG] 1161: cmd_execute_and_stream_data() returned 0 after   334077 us
Stored binaries in 0 destinations in 342 ms.

The indication seems to be "Stored binaries in 0 destinations".

[sohitgore]

+1

[DEBUG] 1241: cmd_execute_and_stream_data() returned 0 after  2325524 us
-- Performing post-build validation
[DEBUG] 1242: execute_process(cd /github_workspace/ebpf_sensor/tools/vcpkg/vcpkg-src/packages/libuuid_x64-linux && zip --quiet -y -r /github_workspace/ebpf_sensor/tools/vcpkg/vcpkg-src/packages/libuuid_x64-linux.zip * --exclude .DS_Store)
[DEBUG] 1242: cmd_execute_and_stream_data() returned 0 after     6649 us
[DEBUG] 1243: execute_process(curl -s -L -H "User-Agent: vcpkg/2024-07-10-d2dfc73769081bdd9b782d08d27794780b7a99b9 (curl)" -H "Accept: application/json;api-version=6.0-preview.1" -H "Content-Type: application/json" -H "Authorization: ***" -X POST --data-raw "***
  \"key\": \"libuuid-320ce6ebe7e8b1732e26ea8df3203987df20257e5a3dd3e2ab2c805632a6f4b0\",
  \"version\": \"320ce6ebe7e8b1732e26ea8df3203987df20257e5a3dd3e2ab2c805632a6f4b0\",
  \"cacheSize\": 33464
***
" https://acghubeus1.actions.githubusercontent.com/IhmGkW4ttwa5pLDWIghNT6yx5tnl9ZxHFwUb2PbUgdTuFh0fXZ/_apis/artifactcache/caches)
[DEBUG] 1243: cmd_execute_and_stream_data() returned 0 after   114622 us
Stored binaries in 0 destinations in 121 ms.

[wench]

Until the underlying issue is fixed, i have modified my workflows to use the action cache to to save and restore the vcpkg_installed directories of my projects. It seems to work well and has eliminated the hour long package builds I was experiencing on every push, Only issue I noticed is that if the version of vcpkg installed in the runner image changes then vcpkg tends to want to rebuild the packages. I decided to add the git hash of the installed vcpkg in the runner image as part of my cache keys so caches are specific for the vcpkg versions. I'm sure there is a better way to do this but it's a good enough solution for me for now.

[andrewward22]

Workaround for manifest mode:

- name: Restore vcpkg cache
  uses: actions/cache/restore@v4
  with:
    path: ~/.cache/vcpkg/archives
    key: some-build-specific-key
    restore-keys: |
      prefix-of-build-specific-key
    
<Run CMake here>

- name: Save vcpkg cache
  uses: actions/cache/save@v4
  with:
    path: ~/.cache/vcpkg/archives
    key: some-build-specific-key

[talregev]

For all the people that subscribe to this issue. vcpkg team decided to remove support in gha. So this issue will be not relevant soon. https://github.com/microsoft/vcpkg-tool/pull/1662

I already see this alternative, https://github.com/microsoft/vcpkg/issues/45073#issuecomment-2828284201 But it not so good as gha binary cache provider was. mainly because if you the cache is invalid, all the needed ports need to be install, instead that only the updated port was installed.

Can we think on other alternative?

[JasonDictos]

Switch to nuget provider, u basically need a PAT classic token and then you're off to the races.

https://github.com/OPM/ResInsight/pull/12407/files

https://docs.github.com/en/packages/learn-github-packages/configuring-a-packages-access-control-and-visibility

Note for those using vcpkg caching from within a docker container you'll have to use a static nuget config pointing to clear text env vars for the token that you can export from a docker secret arg so it is not persistent on the image.

[talregev]

I see your PR. Can you elaborate the json file that you mention here?

      VCPKG_BINARY_SOURCES: "clear;nuget,https://nuget.pkg.github.com/ceetronsolutions/index.json,readwrite"

[ADKaster]

In https://github.com/LadybirdBrowser/ladybird/pull/4461, we went with the "cache the built artifacts" approach, rather than trying to cache $CMAKE_BINARY_DIR/vcpkg_installed/.

The most important part is to set VCPKG_BINARY_SOURCES to the same files source we use for local builds.

• In ./CMakePresets.json, that's "environment": { "VCPKG_BINARY_SOURCES": "clear;files,${fileDir}/Build/caches/vcpkg-binary-cache,readwrite;$penv${VCPKG_BINARY_SOURCES}" },
• In workflows, that's ${{ github.workspace }}/Build/caches/vcpkg-binary-cache (after a checkout action)

Though I can't speak to how good of an idea it is to cache the files source for VCPKG_BINARY_SOURCES. In theory this should do the same thing as the per-package cache entries x-gha was doing, but I also don't know anything about the eviction policies for that vcpkg-tool feature (will it grow indefinitely?) or how well this responds to a vcpkg baseline bump in vcpkg.json (will it see the hash change and force a full rebuild, even though many package versions have not changed?).

[talregev]

we went with the "cache the built artifacts" approach

on the comment I link, it also the cache the built artifacts approach.

In theory this should do the same thing as the per-package cache entries x-gha was doing.

It not the same. when you invalidate the cache, all the packages need to rebuild. on x-gha, no need to invalidate the cache (you don't have a key for that also), only the updated packages was updating, and if i don't mistaken in the x-gha, after 30 days that the packages didn't restore (due it old), then it automatics deleted.

[JasonDictos]

I see your PR. Can you elaborate the json file that you mention here?

      VCPKG_BINARY_SOURCES: "clear;nuget,https://nuget.pkg.github.com/ceetronsolutions/index.json,readwrite"

Not my pr; I work on a closed repo, that was what I used as a guide though.

That json file isn't something you edit thats just the endpoint that comes with the github product for stashing nuget packages. If you're building directly in github actions that pr has everything you need, however we build within docker so we had to forward the token into the container using a secret like so:

else ifdef GH_PACKAGES_TOKEN
image: Dockerfile
  DOCKER_BUILDKIT=1 docker build -t $(IMAGE_PATH) . -f Dockerfile \
          --secret id=gh_pkg_token,env=GH_PACKAGES_TOKEN\
          --secret id=gh_token,env=BOT_ACCESS_TOKEN --progress plain
else

Then in the docker build step:

# Install pre-packaged ports for the image
RUN --mount=type=ssh \
    --mount=type=secret,id=gh_pkg_token vcpkg ....

And somewhere you have to set the binary sources env:

    export VCPKG_BINARY_SOURCES="clear;nuget,GitHubPackages,readwrite"

You will need to install mono-complete and dotnet-sdk-9.0, and then nuget will work by prefixing it with 'mono' (vcpkg fetch nuget will download nuget.exe itself).

The way we got around persistently storing the token in the docker image is we used a config template:

COPY resource/NuGet.Config /root/.config/NuGet/NuGet.Config

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
    <add key="GitHubPackages" value="https://nuget.pkg.github.com/%GH_PACKAGES_USER%/index.json" />
  </packageSources>
  <packageSourceCredentials>
    <GitHubPackages>
        <add key="Username" value="%GH_PACKAGES_USER%" />
        <add key="ClearTextPassword" value="%GH_PACKAGES_TOKEN%" />
      </GitHubPackages>
  </packageSourceCredentials>
  <apikeys>
    <add key="https://nuget.pkg.github.com/%GH_PACKAGES_USER%/index.json" value="%GH_PACKAGES_TOKEN%" />
  </apikeys>
</configuration>        

So by loading the secret from /run/secrets/gh_pkg_token into the GH_PACKAGES_TOKEN_ENV, and by setting your github user name env GH_PACKAGES_USER nuget will authenticate...

[talregev]

@JasonDictos Thank you for your detail comment! I still want to understand more about the nuget server.

https://nuget.pkg.github.com/ceetronsolutions/index.json

This is a public server? It not match the user nor the repo-> OPM/ResInsight

Should I open a nuget server? or if I have a user in gha, let say talregev, should I replace the https://nuget.pkg.github.com/talregev/index.json

and where the user magnesj came from? taken from: ${{ env.USERNAME }}

And as I understand this parameter is build in inside gha: ${{ secrets.GH_PACKAGES_TOKEN }}

Thank you for your help.

[JasonDictos]

If your GitHub url is

GitHub.com/MyName/project

Then USERNAME is MyName

So in this case: https://nuget.pkg.github.com/ceetronsolutions/index.json

Username is ceetronsolutions

Everything else is the same.

For the token is is generated from a personal access token (classic) it is documented here: https://learn.microsoft.com/en-us/vcpkg/consume/binary-caching-github-actions-cache

[HunterZ]

Can we think on other alternative?

I ended up using actions/cache@v4 with the following key setup (I use CMake manifest mode) which hashes over anything that seems like it might affect validity of the vcpkg build products:

          key:
            cache-build-${{ matrix.triplet }}-${{ hashFiles('CMakeFiles.txt', 'CMakePresets.json', 'vcpkg.json', 'out/build/*/vcpkg_installed/compiler-file-hash-cache.json', 'out/build/*/vcpkg_installed/status') }}
          restore-keys: |
            cache-build-${{ matrix.triplet }}-

[magnesj]

@JasonDictos Thank you for your detail comment! I still want to understand more about the nuget server.

https://nuget.pkg.github.com/ceetronsolutions/index.json

This is a public server? It not match the user nor the repo-> OPM/ResInsight

Should I open a nuget server? or if I have a user in gha, let say talregev, should I replace the https://nuget.pkg.github.com/talregev/index.json

and where the user magnesj came from? taken from: ${{ env.USERNAME }}

And as I understand this parameter is build in inside gha: ${{ secrets.GH_PACKAGES_TOKEN }}

Thank you for your help.

Related links https://docs.github.com/en/packages/working-with-a-github-packages-registry/working-with-the-nuget-registry https://docs.github.com/en/billing/managing-billing-for-your-products/managing-billing-for-github-packages/about-billing-for-github-packages

The NuGet packages are hosted at the top-level GitHub page (not on inside a repository) either on a personal or organization site. Here is a snapshot from the packages available at the Packages tab from my CeetronSolutions organization site.

[JasonDictos]

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <packageSources>
    <add key="nuget.org" value="https://api.nuget.org/v3/index.json" protocolVersion="3" />
    <add key="GitHubPackages" value="https://nuget.pkg.github.com/%GH_PACKAGES_USER%/index.json" />
  </packageSources>
  <packageSourceCredentials>
    <GitHubPackages>
        <add key="Username" value="%GH_PACKAGES_USER%" />
        <add key="ClearTextPassword" value="%GH_PACKAGES_TOKEN%" />
      </GitHubPackages>
  </packageSourceCredentials>
  <apikeys>
    <add key="https://nuget.pkg.github.com/%GH_PACKAGES_USER%/index.json" value="%GH_PACKAGES_TOKEN%" />
  </apikeys>
</configuration>        
So by loading the secret from /run/secrets/gh_pkg_token into the GH_PACKAGES_TOKEN_ENV, and by setting your github user name env GH_PACKAGES_USER nuget will authenticate...

I'm sorry but this doesn't work; problem is the NugGet config expects the value for the apikey to be an encrypted value, this causes this error, you will have to authenticate once and have it encrypt it to update the config:

nuget setapikey $GH_PACKAGES_TOKEN -Source https://nuget.pkg.github.com/%GH_PACKAGES_USER%/index.jso

That will set the api key to an encrypted value...

[magnesj]

@jackjansen vcpkg maintainer suggests NuGet as cache, see https://github.com/curl/curl/pull/17200#issuecomment-2839854264

[rustyconover]

@jackjansen vcpkg maintainer suggests NuGet as cache, see curl/curl#17200 (comment)

For anyone exploring this approach, please note that accessing the NuGet cache via vcpkg currently requires a compatible Mono (C# runtime) installation.

It would be great to see support for a lightweight, statically linked alternative to Mono or the AWS CLI for interacting with cache APIs.

Additionally, improved native integration of cache functionality within vcpkg — avoiding frequent external process spawning — could help performance, especially in CPU-constrained CI environments where process creation can become a bottleneck.

[JasonDictos]

@rustyconover agreed! Our image went from 800MB to 1.1GB just to support NuGet/mono-complete/dotnet-sdk-9 on our Fedora docker images...

[Derpius]

Not what I'd call "production ready", but we've written a lightweight action which leverages the built-in filesystem binary cache provider and @actions/cache to facilitate per-package caching in GHA: https://github.com/TAServers/vcpkg-cache

Didn't want to add a dependency on NuGet just to cache dependencies

[talregev]

Not what I'd call "production ready", but we've written a lightweight action which leverages the built-in filesystem binary cache provider and @actions/cache to facilitate per-package caching in GHA: https://github.com/TAServers/vcpkg-cache

Didn't want to add a dependency on NuGet just to cache dependencies

Cool Idea! Can you add an example how to use it?

[jackjansen]

@Derpius that would be very nice... I really can't be bothered to learn NuGet and all that just to get vcpkg caching working...