Annotations are not set to EphemeralRunner pods

[andrey-petrenko-develeap]

Checks

• [x] I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• [x] I am using charts that are officially provided

Controller Version

0.11.0

Deployment Method

Helm

Checks

• [x] This isn't a question or user support case (For Q&A and community support, go to Discussions).
• [x] I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1. Install `actions-runner-controller` and `gha-runner-scale-set` Helm charts with Terraform `helm_release` resource
2. Update `gha-runner-scale-set` values with:

annotations:
  ad.datadoghq.com/exclude: "true"

3. Update `gha-runner-scale-set` Helm release with Terraform
4. Annotation is added to `AutoscalingRunnerSet` resource, but any EphemeralRunner` is created without it

Describe the bug

I am installing actions-runner-controller and gha-runner-scale-set Helm charts with Terraform. I need to add an annotation ad.datadoghq.com/exclude: "true" to all EphemeralRunner pods. I set it in the values file, and after the apply, any EphemeralRunner is still created without it.

Now looking again at values.yaml at the chart repo I see the line "## Optional annotations and labels applied to all resources created by helm installation" which would make sense, as EphemeralRunners are not directly created by Helm.

But then I'd expect all the custom annotations being applied to the downstream resources, controlled by the ones, that were created with Helm, i.e. AutoscalingRunnerSet -> EphemeralRunnerSet -> EphemeralRunner

Describe the expected behavior

Annotations that are set at annotations: section of the Helm values should propagate to every created resource, as documented, or there should be another way of setting runner pod annotations with Helm chart.

Additional Context

Terraform code for scale set Helm release:

resource "helm_release" "runners_aws_4g_scale_set" {
  count = var.github_runners_enabled ? 1 : 0

  name       = "runner-aws-${var.aws_profile}-4g"
  namespace  = local.github_runners_namespace
  repository = "oci://ghcr.io/actions/actions-runner-controller-charts"
  chart      = "gha-runner-scale-set"
  version    = "0.11.0"

  depends_on = [helm_release.actions_runner_controller]

  values = [
    file("${path.module}/../runners.yaml"),
  ]
  set {..}

}


Helm values after apply:

helm -n github-runners get values runner-aws-dev-4g

USER-SUPPLIED VALUES:
affinity: {}
annotations:
  ad.datadoghq.com/exclude: "true"
containerMode:
  type: dind
githubConfigSecret:
[...]
labels: {}
maxRunners: 16
minRunners: 1
podLabels: {}
podSecurityContext: {}
priorityClassName: ""
resources:
  limits:
    cpu: 100m
    memory: 128Mi
  requests:
    cpu: 100m
    memory: 128Mi
runnerGroup: default
runnerLabels: []
securityContext: {}
serviceAccount:
  create: true
  name: gha-runner-scale-set
template:
  spec:
    containers:
    - command:
      - /home/runner/run.sh
      image: *redacted*/runner:0.1.2
      imagePullPolicy: IfNotPresent
      name: runner
      resources:
        limits:
          memory: 4Gi
      volumeMounts:
      - mountPath: /runner/_work
        name: runner-cache
    imagePullSecrets:
    - name: github-token
      namespace: github-runners
    volumes:
    - emptyDir: {}
      name: runner-cache
topologySpreadConstraints: []
volumeMounts: []
volumes: []


AutoscalingRunnerSet CRD after apply - the annotation is in the list:

kubectl describe AutoscalingRunnerSet runner-aws-dev-4g -n github-runners
Name:         runner-aws-dev-4g
Namespace:    github-runners
Labels:       actions.github.com/organization=***
[...]
              app.kubernetes.io/part-of=gha-rs
              app.kubernetes.io/version=0.11.0
              helm.sh/chart=gha-rs-0.11.0
Annotations:  actions.github.com/cleanup-github-secret-name: runner-aws-dev-4g-gha-rs-github-secret
              actions.github.com/cleanup-manager-role-binding: runner-aws-dev-4g-gha-rs-manager
              actions.github.com/cleanup-manager-role-name: runner-aws-dev-4g-gha-rs-manager
              actions.github.com/cleanup-no-permission-service-account-name: runner-aws-dev-4g-gha-rs-no-permission
              actions.github.com/runner-group-name: Default
              actions.github.com/runner-scale-set-name: runner-aws-dev-4g
              actions.github.com/values-hash: 98612cfa3c2e56aec9d889a91a633ca6c455c80df366d637ad800e27c9aba5c
              ad.datadoghq.com/exclude: true
              meta.helm.sh/release-name: runner-aws-dev-4g
              meta.helm.sh/release-namespace: github-runners
              runner-scale-set-id: 26
API Version:  actions.github.com/v1alpha1
Kind:         AutoscalingRunnerSet
[...]
Status:
  Current Runners:            1
  Pending Ephemeral Runners:  1
Events:                       <none>


Example of the runner pod:

kubectl describe pod runner-aws-dev-4g-f8q97-runner-ljvlv -n github-runners

Name:             runner-aws-dev-4g-f8q97-runner-ljvlv
Namespace:        github-runners
Priority:         0
Service Account:  runner-aws-dev-4g-gha-rs-no-permission
Node:             ***
Start Time:       Tue, 24 Jun 2025 14:20:13 +0300
Labels:           actions-ephemeral-runner=True
[...]
                  app.kubernetes.io/name=runner-aws-dev-4g
                  app.kubernetes.io/part-of=gha-runner-scale-set
                  app.kubernetes.io/version=0.11.0
                  helm.sh/chart=gha-rs-0.11.0
                  pod-template-hash=5dbbbc8d65
Annotations:      actions.github.com/patch-id: 0
                  actions.github.com/runner-group-name: Default
                  actions.github.com/runner-scale-set-name: runner-aws-dev-4g
                  actions.github.com/runner-spec-hash: 5ffc57fc6
                  cluster-autoscaler.kubernetes.io/safe-to-evict-local-volumes: datadog
Status:           Running
[...]
Controlled By:  EphemeralRunner/runner-aws-dev-4g-f8q97-runner-ljvlv
[...]
Conditions:
  Type                        Status
  PodReadyToStartContainers   True 
  Initialized                 True 
  Ready                       True 
  ContainersReady             True 
  PodScheduled                True 
[...]
Events:
[...]

Controller Logs

no relevant logs

Runner Pod Logs

no relevant logs

[github-actions[bot]]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[igor-sidorov]

Having same issue. Annotations from scale set not set to ephemeral runners.

[jgutierrezglez]

This issue is also impacting us, and a resolution is critical since we need to annotate the runners in order to prevent eviction

[ejstreet]

There's currently no way to do it in the chart, but as a workaround you can use the ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE env var to mount a configmap that contains a template.

Here's an example that adds an annotation and a GPU:

# values.yaml
template:
  spec:
    containers:
      - name: runner
        env:
          - name: ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE
             value: /home/runner/pod-templates/gpu-runner-template.yaml
        volumeMounts:
          - name: work
            mountPath: /home/runner/_work
          - name: pod-templates
            mountPath: /home/runner/pod-templates
            readOnly: true
    volumes:
      - name: pod-templates
        configMap:
          name: gpu-runner-pod-template

And then the configmap:

apiVersion: v1
kind: ConfigMap
metadata:
  name: gpu-runner-pod-template
  namespace: arc-runners
data:
  gpu-runner-template.yaml: |
    ---
    metadata:
      annotations:
        karpenter.sh/do-not-disrupt: "true"
    spec:
      containers:
      - name: $job
        resources:
          limits:
            nvidia.com/gpu: 1
      securityContext:
        fsGroup: 123

These values get merged with the other values generated by the runner.

[jgutierrezglez]

Hello

There's currently no way to do it in the chart, but as a workaround you can use the ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE env var to mount a configmap that contains a template.

Here's an example that adds an annotation and a GPU:

values.yaml

template: spec: containers: - name: runner env: - name: ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE value: /home/runner/pod-templates/gpu-runner-template.yaml volumeMounts: - name: work mountPath: /home/runner/_work - name: pod-templates mountPath: /home/runner/pod-templates readOnly: true volumes: - name: pod-templates configMap: name: gpu-runner-pod-template And then the configmap:

apiVersion: v1 kind: ConfigMap metadata: name: gpu-runner-pod-template namespace: arc-runners data: gpu-runner-template.yaml: | --- metadata: annotations: karpenter.sh/do-not-disrupt: "true" spec: containers: - name: $job resources: limits: nvidia.com/gpu: 1 securityContext: fsGroup: 123 These values get merged with the other values generated by the runner.

Hello @ejstreet,

I tried your proposed workaround to include the annotations and I cannot make it work..

My configmap (I'm not interested in the GPU, so, I'm removing it):

apiVersion: v1
kind: ConfigMap
metadata:
  name: runner-pod-template
  namespace: {{ .Release.Namespace }}
data:
  runner-template.yaml: |
    ---
    metadata:
      annotations:
        karpenter.sh/do-not-disrupt: "true"
    spec:
      securityContext:
        fsGroup: 123

# values.yaml
template:
  spec:
    containers:
      - name: runner
        env:
          - name: ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE
             value: /home/runner/pod-templates/runner-template.yaml
        volumeMounts:
          - name: work
            mountPath: /home/runner/_work
          - name: pod-templates
            mountPath: /home/runner/pod-templates
            readOnly: true
    volumes:
      - name: pod-templates
        configMap:
          name: runner-pod-template

I can see the content of the configMap properly mounted inside the runner, but no values get merged. I can also confirm that I'm using ephemeral runner

Do you know what could be the issue? Or where is the location of the logs related to execution of the hook?

Regards,

Javier G.

[ejstreet]

Ah, sorry, it should go under .Values.template.metadata.annotations, as per here.

ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE applies for kubernetes mode workflows, apologies for the mix up.

[jgutierrezglez]

ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE

In our case, we're using dind, do you know if there a way to make it work in this mode? I couldn't find any mention to this limitation:

ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE applies for kubernetes mode workflows

Could you share a pointer to the documentation where it's mentioned?

Regards,

Javier G.

[95jinhong]

I declared the following in the YAML file, and I confirmed that on the node where the pod is deployed, consolidation does not occur because of this pod.

spec: 
  template:
    metadata: 
      annotations: 
        karpenter.sh/do-not-disrupt: 'true'

I hope this helps. Thank you. Hong

[jgutierrezglez]

@95jinhong that works ! Thank you so much !

[nikola-jokic]

Hey, the ephemeral runner pod spec is completely in your control. The controller applies few things, but we didn't want to apply annotations that might influence the pod in an unintended way, while you can apply desired modifications in a way that @95jinhong described (thank you for doing that) ☺️ Closing this one since it is working as intended. We might change this in the future, but for now, this is exactly the intended behavior.