runner-images icon indicating copy to clipboard operation
runner-images copied to clipboard

Fail to hash files

Open adoroszlai opened this issue 1 month ago • 25 comments

Description

Workflow fails with error:

The template is not valid. ... hashFiles('...') failed. Fail to hash files under directory '/Users/runner/work/...'

The same step works fine both on previous version of macos-15 runners and on ubuntu-24.04.

Platforms affected

  • [ ] Azure DevOps
  • [x] GitHub Actions - Standard Runners
  • [ ] GitHub Actions - Larger Runners

Runner images affected

  • [ ] Ubuntu 22.04
  • [ ] Ubuntu 24.04
  • [ ] Ubuntu Slim
  • [ ] macOS 13
  • [ ] macOS 13 Arm64
  • [ ] macOS 14
  • [ ] macOS 14 Arm64
  • [x] macOS 15
  • [x] macOS 15 Arm64
  • [ ] macOS 26 Arm64
  • [ ] Windows Server 2019
  • [ ] Windows Server 2022
  • [ ] Windows Server 2025

Image version and build link

Broken:

  • macos-15-arm64 version 20251119.0020: https://github.com/adoroszlai/ozone/actions/runs/19597747914/job/56125544615
  • macos-15 (Intel) version 20251120.0023: https://github.com/adoroszlai/ozone/actions/runs/19597747914/job/56124691970

Works:

  • macos-15-arm64 version 20251104.0104: https://github.com/adoroszlai/ozone/actions/runs/19597747914/job/56124691976
  • macos-15 (Intel) version 20251103.0112: https://github.com/adoroszlai/ozone/actions/runs/19597747914/job/56125544610

Is it regression?

yes

Expected behavior

hashFiles() calculates hash of files

  • https://github.com/adoroszlai/ozone/actions/runs/19597747914/job/56124691976
  • https://github.com/adoroszlai/ozone/actions/runs/19597747914/job/56125544610

Actual behavior

Error: The template is not valid. ... hashFiles('...') failed. Fail to hash files under directory '...'

  • https://github.com/adoroszlai/ozone/actions/runs/19597747914/job/56125544615
  • https://github.com/adoroszlai/ozone/actions/runs/19597747914/job/56124691970

Repro steps

See repro workflow: https://github.com/adoroszlai/ozone/actions/runs/19597747914/workflow

adoroszlai avatar Nov 22 '25 16:11 adoroszlai

Caused by duplicated lines in index.js: https://github.com/orgs/community/discussions/180160#discussioncomment-15047581

andrzejressel avatar Nov 22 '25 16:11 andrzejressel

Same on Tahoe

jdrozdnovak avatar Nov 22 '25 18:11 jdrozdnovak

Also affects macos-14

sjp4 avatar Nov 22 '25 18:11 sjp4

We’re running into this bug at Pion https://github.com/pion/webrtc/pull/3275

JoeTurki avatar Nov 22 '25 21:11 JoeTurki

As a workaround on one project, we just temporarily disabled caching for macOS:

https://github.com/babashka/fs/pull/170/files#diff-b803fcb7f17ed9235f1e5cb1fcd2f5d3b2838429d4368ae4c57ce4436577f03fR35-R38

lread avatar Nov 22 '25 21:11 lread

Experiencing the same issue.

Vyazovoy avatar Nov 23 '25 06:11 Vyazovoy

@Alexey-Ayupov @erik-bershel Not sure who to ping about this problem. If it cannot be fixed soon, please rollback to the previous version, which is known to work OK.

adoroszlai avatar Nov 23 '25 10:11 adoroszlai

I am also being affected by this: https://github.com/drupal/cms-launcher/actions/runs/19608154195/job/56149953137#step:7:1

phenaproxima avatar Nov 23 '25 12:11 phenaproxima

This has also broken our CI/CD! Thank you in advance to whoever can get this resolved ASAP.

code-ape avatar Nov 23 '25 13:11 code-ape

+1

thomhurst avatar Nov 23 '25 14:11 thomhurst

same on arm64 macos-26: https://github.com/user4223/ticket-decoder/actions/runs/19601406315/job/56133538243

user4223 avatar Nov 23 '25 14:11 user4223

Note that some runner instances may not be affected (yet) - my projects trigger two macos-15 builds in parallel (without any build image version pinning) but may only fail one of them while the other is still able to come up with a valid hash.

dneuge avatar Nov 23 '25 14:11 dneuge

Same here! macOS 26

izamontenegro avatar Nov 23 '25 15:11 izamontenegro

Same Here

tomasmark79 avatar Nov 23 '25 15:11 tomasmark79

Same in https://github.com/swift-dns/swift-endpoint/

MahdiBM avatar Nov 23 '25 15:11 MahdiBM

appears that even hashFiles with multiple globs is also borked

robtaylor avatar Nov 23 '25 17:11 robtaylor

appears to be related to https://github.com/actions/runner/commit/7df164d2c7c2f5f2207d4a74c273c3c1d183f831 @TingluoHuang

robtaylor avatar Nov 23 '25 17:11 robtaylor

Seeing this as well.

SeanTAllen avatar Nov 24 '25 00:11 SeanTAllen

Thank you all for the reports. We have confirmed the issue with the latest macOS image versions and have initiated a rollback to the previous versions.

EricHorton avatar Nov 24 '25 01:11 EricHorton

All images have been rolled back to the previous stable version. Updates have been suspended pending the investigation. The issue will remain open until the upstream repository is fully resolved.

erik-bershel avatar Nov 24 '25 11:11 erik-bershel

I've implemented a fix for what appears to be the same issue in the Jamulus macOS builds https://github.com/jamulussoftware/jamulus/pull/3565#issuecomment-3568870379

pljones avatar Nov 24 '25 18:11 pljones

We've identified the cause of the issue on the service side. The team responsible is working to resolve it. Additionally, a working temporary patch has been found on our side, which will be applied before the upcoming update in order to unblock image updates while fixing root cause. A recurrence is not expected in any way.

I'd like to point out that there's no need to apply patches to the user workflows - all affected runners have been restored to a working state.

To raise awareness, I'll keep this issue open until the problem on the service side is fully resolved.

UPD: As a separate note, self-hosted runners built using our code are not susceptible to this problem, as it resides outside the image.

erik-bershel avatar Nov 24 '25 19:11 erik-bershel

I think this issue can now be closed?

jamietanna avatar Dec 12 '25 09:12 jamietanna

It was mentioned by @erik-bershel https://github.com/actions/runner-images/issues/13341#issuecomment-3572431071 that further work was needed "on the service side" -- no one's posted to confirm that work has been completed.

pljones avatar Dec 12 '25 18:12 pljones

We are awaiting confirmation.

erik-bershel avatar Dec 15 '25 09:12 erik-bershel

Root cause was eliminated. Closing as completed.

erik-bershel avatar Dec 22 '25 10:12 erik-bershel