runner icon indicating copy to clipboard operation
runner copied to clipboard

The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.

Open HarshadDGhorpade-eaton opened this issue 10 months ago • 2 comments

Describe the bug Time to time we're getting this error and workflow is getting canceled due to that, its very frustating. We have looked into couple of other issues as well: The runner has received a shutdown signal. · Issue #6709 · actions/runner-images

The runner has received a shutdown signal. · actions/runner-images · Discussion #7188

we have reduced number of threads needed for our build to 28 from total capacity of 32 but still its same.

Runner Version and Platform

Runner version : '2.322.0' We're using larger runner with 32 cores Image: ubuntu-22.04 Version: 20250209.1.0

OS of the machine running the runner? OSX/Windows/Linux/... OS : Ubuntu 22.04.5

What's not working?

The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
Process completed with exit code 137.
The operation was canceled.

HarshadDGhorpade-eaton avatar Feb 24 '25 05:02 HarshadDGhorpade-eaton

We are facing the same issue and have conducted several debugging steps:

  1. Increased Resources: Upgraded runners from 4 CPU / 16GB RAM to 8 CPU / 32GB RAM—no improvement.
  2. Resource Starvation: No signs of CPU/memory exhaustion; Grafana shows nodes staying well below 10% utilization.
  3. Network Stability: No network disruptions observed; runner connects successfully but stops unexpectedly after a few minutes.
  4. Logs: No meaningful logs indicating the cause of disconnection or failure.
  5. Runner Status: Manually connecting to the runner shows it functioning normally, not stuck or unresponsive.
  6. Version Downgrade: Rolling back from 2.322.0 to 2.321.0 had no effect.
  7. Kubernetes Setup: Running on Kubernetes using the new GitHub Actions Runner Controller 0.10.2 (migrated from summerwinds).
  8. OS Upgrade Suspicions: We suspect a potential issue with the Ubuntu 24.04 upgrade, but no confirmation yet.
  9. Custom Docker Image: Based on the official GitHub Actions runner image, adding necessary tools on top.
  10. GitHub API Rate Limits: No signs of hitting rate limits; GitHub.com statistics show no related errors.
  11. Time-Based Pattern?: Failure happens more often at night (US time), but this is an observation, not a confirmed cause.

Any insights or guidance on further debugging steps would be greatly appreciated!

matanbaruch avatar Feb 26 '25 09:02 matanbaruch

i can tell you i have this issue in EKS , Ec2 or any build i try to do from time to time i get the issue is from ubuntu 22xx,24xx

my test was on ec2 32gig 8core and same errors not much help there i try to compile the agent , build my one one , use github compile one but same results

"The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled"

not sure why this is know issues from 2022 that not being fixed

noamgreen avatar Mar 17 '25 07:03 noamgreen

we have the same issue, but seems likely its killed by other process. i am trying to see if i can reserve some resources for github runner , so that it wont be killed.

zoezhangmattr avatar Apr 01 '25 22:04 zoezhangmattr

Just wanted to chime in - it seems this issue still persists on my side. https://github.com/aurora-is-near/borealis-engine-lib/actions/runs/15438833121/job/43451560080

I’m getting the exact same message: The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.

Has anyone found a reliable workaround or fix for this? Would appreciate any tips or insight.

alexander-borodulya avatar Jun 04 '25 11:06 alexander-borodulya

Just wanted to chime in - it seems this issue still persists on my side. https://github.com/aurora-is-near/borealis-engine-lib/actions/runs/15438833121/job/43451560080

I’m getting the exact same message: The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.

Has anyone found a reliable workaround or fix for this? Would appreciate any tips or insight.

Hey man I see it works fine for you now, did you find a workaround ? I suddenly have the same issue for about a week now.

benjamin-rousseau-shift avatar Jun 25 '25 11:06 benjamin-rousseau-shift

@benjamin-rousseau-shift A possible workaround is to switch to selfhosted runner instead of GitHub-hosted (Not the perfect solution, but works for me atm) https://github.com/aurora-is-near/borealis-engine-lib/pull/230/files#diff-73e17259d77e5fbef83b2bdbbe4dc40a912f807472287f7f45b77e0cbf78792dR87

alexander-borodulya avatar Jun 26 '25 11:06 alexander-borodulya

Not sure if this is the root cause and solution but posting it here for someone to validate.

Root Cause:

It could be due to the Linux kernel inside the VM when it hits an out-of-memory (OOM) limit. GitHub then kills the whole job and the runner reports the generic message you saw.

Found an article here which mentions the RAM and storage limits for GitHub hosted runners: https://blog.jakelee.co.uk/gradle-build-freezing-on-github-ci/#cause--resolution

Solutions could include;

[1] Capping RAM and storage used [2] Using GitHub Large Runners or the workaround mentioned by @alexander-borodulya by using a self-hosted runner.

pritesh-0409 avatar Jun 26 '25 14:06 pritesh-0409