Windows Container for Action Runner

[ttasharski73]

Is a windows container image supported for the action-runner-controller system? I am able to get Linux working fine, but we want the ability to do both Linux and Windows.

[mumoshu]

@ttasharski73 Hey! Unfortunately, it isn't supported today. I don't even know- is self-hosted runner agent supposed to run on Windows?

[ttasharski73]

@mumoshu if I could do just Linux runners I would, but need some Windows runners for builds that can only be done on Windows. Github offers Windows hosted runners, but I would prefer to run self-hosted and on my AKS. I was just seeing if anyone has experience with it, if not I will try to figure it out.

[mumoshu]

@ttasharski73 Thanks for the clarification! I think what you would need to do would be:

• Build a Windows container image(is there anything like that?) that looks like a Windows version of https://github.com/actions-runner-controller/actions-runner-controller/blob/master/runner/Dockerfile and https://github.com/actions-runner-controller/actions-runner-controller/blob/master/runner/entrypoint.sh
• Point RunnerDeployment.Spec.Template.Spec.RunnerImage to your custom docker image

Probably https://github.com/actions/virtual-environments/tree/main/images/win can be your source of inspiration when building your Windows container image for runners.

[ttasharski73]

@mumoshu thanks! I will give it a try! I think I can use a Windows 2019 base image that what it looks like Github is using.

[isarkis]

I was able to get windows based runner working in GKE. Had to make changes in actions-runner-controller. Not sure about other Kubernetes services, but GKE wouldn't admit a windows based pod with 'privileged=true'.

Is there a particular reason actions-runner-controller sets 'prileged=true' even with 'dockerEnabled: false' ? I would imagine it would be useful to allow users make their self hosted runners run in unprivileged mode. This might be a good option for those who want to harden security around running self hosted runner, albeit at the expense of not being able to run container based workflows in K8s.

[mumoshu]

@isarkis Thanks for sharing your experience!

I imagine there would be cases that you use docker in workflows but without privileges, e.g. you use kata containers or sysbox containers to make dind work without privileged containers.

That said, I think adding privileged: [true|false] option to runner and runnerdeployment specs seem to be a valid approach, which enough caution to users! Would you mind submitting a pull request based on your work?

[isarkis]

According to the readme, setting 'dockerEnabled: false' should create unprivileged container, but it doesn't appear to be doing so:

# true (default) = A privileged docker sidecar container is included in the runner pod.
# false = A docker sidecar container is not included in the runner pod and you can't use docker.
# If set to false, there are no privileged container and you cannot use docker.
dockerEnabled: false

I am thinking of just fixing that instead of adding another mutually exclusive option (i.e. 'privileged'). What do you think?

BTW. I was able to get the actions-runner-controller working with sysbox, that should make dind workflows more secure.

[mumoshu]

@isarkis Good catch! Yeah, that should be enough for this specific issue.

adding another mutually exclusive option (i.e. 'privileged')

I think the privileged option would be still handy, because it can be used to enable docker while disabling privilege, which is impossible with dockerEnabled alone.

In the end you'd use:

1. privileged: false + dockrEnabled: true for sysbox (i.e. dockerEnabled: true implies privileged: true but you explicitly negate privileged with `privileged: false
2. dockerEnabled: false for Windows containers on GKE (i.e. dockerEnabled: false implies privileged: false, if that's fine for you

BTW. I was able to get the actions-runner-controller working with sysbox, that should make dind workflows more secure.

Good news! Thanks for sharing ☺️

[sbrinkerhoff]

How have folks managed with Windows runners? What is working, what isn't? Any overview of how to get it setup?

[GoranHalvarsson]

I would also like to know this. How to setup windows runners with actions-runner-controller? Is this possible today?

[GoranHalvarsson]

Hello @isarkis :-) Could please share how you got windows runners working in actions-runner-controller.

[isarkis]

Unfortunately, I haven't had any time to create proper PR. Also, I got this working on our heavily customized GKE cluster which is different from vanilla GKE, so your mileage might vary.

Having said that, I got windows runner working by taking v0.20.2 and making some changes to controller. The controller change was necessary to get windows runner pod admitted to our cluster, this might not be necessary for your cluster, but I am not sure. My GKE cluster rejects windows pods with security related settings - these simply aren't supported by Kubernetes. Most Kubernetes clusters will just ignore these settings, but my GKE cluster simply won't admit it. Keep in mind that with this change in place Linux runners won't work.

You can get the code from my fork and here is the change.

To get this working I had the following done:

• Build controller from my fork and deploy.
• Build windows runner image from the Dockerfile and push to registry.
• Create Runner deployment using the windows runner image. Something along these lines:

apiVersion: actions.summerwind.dev/v1alpha1
kind: Runner
metadata:
  name: windows-runner
spec:
  nodeSelector:
    kubernetes.io/os: windows
  repository: XXXXXX
  image: gcr.io/XXXXX/windows-runner:latest
  dockerEnabled: false
  env: []

[antgrutta]

I too have been working on this problem, Windows Runner Container. I ended up not having to make any changes to the actions-runner-controller code as I was using containerd on my Windows nodepool. I am going to further refine my entrypoint.ps1 to account for the different switches beyond what my POC used (organization based runners). @isarkis would you like to combine efforts and maybe put up a PR together?

[isarkis]

@antgrutta, did you try your windows runners on GKE or some other Kubernetes cluster?

[antgrutta]

@antgrutta, did you try your windows runners on GKE or some other Kubernetes cluster?

I am testing this in Azure using AKS, sorry should have added that detail first.

[GoranHalvarsson]

Thank you @isarkis for your fork! This is great news!!

I also run on AKS @antgrutta. So would it work to just point to a windows image in the runner deployment:

apiVersion: actions.summerwind.dev/v1alpha1
kind: Runner
metadata:
  name: windows-runner
spec:
  nodeSelector:
    kubernetes.io/os: windows
  repository: XXXXXX
  image:  mcr.microsoft.com/windows/servercore:ltsc2019
  dockerEnabled: false
  env: []

Could this work on AKS?

But we still need to call the script CMD ["powershell", ".\runner.ps1"]

Is that what you are doing using the entrypoint.ps1, @antgrutta ?

[GoranHalvarsson]

Let me update my own comment :-) It worked great running a windows runner. Thank you @isarkis :-) I did not have to use your fork, @isarkis . I used however your dockerfile to create the windows image. So thank you again :-)

[antgrutta]

Is that what you are doing using the entrypoint.ps1, @antgrutta ?

Yes, it provides a mechanism to update switches based on the incoming environment variables from the controller.

[shinji62]

@isarkis Any chance to publish the sysbox implementation that will be more than welcome. ?

[isarkis]

@shinji62, here is what I did:

• Create new GKE cluster version 1.21.5-gke.1302 using Ubuntu with containerd (ubuntu_containerd) image. Review k8s distro requirements and cluster setup notes as they are different for each k8s provider. There is also k8s version requirement.
  • Add 'sysbox-install: yes' kubernetes label in Metadata (this will make sure that all existing and new nodes get Sysbox installed)
  • Do NOT enable secure-boot (might be GKE specific).
• Follow these directions to install Sysbox on your cluster.
• Install Actions Runner Controller.
• Follow Sysbox Pod Deployment instructions to deploy your runners. Here is how my deployment looks like:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: linux-runner
  namespace: default
spec:
  replicas: <NUMBER_OF_RUNNERS>
  template:
    metadata:
      annotations:
        io.kubernetes.cri-o.userns-mode: "auto:size=65536"
    spec:
      repository: <YOUR_REPO>
      labels:
        - self-hosted
      env:
        # Tell controller to use --ephemeral vs to be deprecated --once flag.
        - name: RUNNER_FEATURE_FLAG_EPHEMERAL
          value: "true"
        # Delay start to avoid hitting
        # https://github.com/actions-runner-controller/actions-runner-controller/issues/787
        - name: STARTUP_DELAY_IN_SECONDS
          value: "5"
      runtimeClassName: "sysbox-runc"
      imagePullPolicy: Always
      ephemeral: true
      dockerdContainerResources:
        limits:
          cpu: "16"
          memory: "64Gi"
        requests:
          cpu: "10"
          memory: "32Gi"

Keep in mind, Sysbox CE has certain limitations. For instance, number of pods per node is capped at 16.

[philqa]

Hi @GoranHalvarsson @antgrutta @isarkis - I have Windows runners working in AWS EKS using a similar Docker image and a few cluster modifications. However the HorizontalRunnerAutoscaler for the windows-runners isn't working, is this working for you?

[ttasharski73]

@philqa I was not able to get the Run Autoscaler to work with the windows runner, I just spun a few static windows runners managed by the action runner.

[isarkis]

Same here, only tried static windows runner.

[philqa]

Thanks for your replies @ttasharski73, @isarkis. I managed to get this working in EKS after re-reading the comments above/your forks, creating and deploying my own actions-controller and a few more minor modifications.

[mazilu88]

I was able to run on AKS with no issues. When configuring the runner, I used a PowerShell script that I try to keep up to date with this https://github.com/actions-runner-controller/actions-runner-controller/blob/master/runner/startup.sh

if (-not (Test-Path env:RUNNER_NAME)) {
    Write-Host "RUNNER_NAME must be set"
    exit 1
}

if (-not (Test-Path env:RUNNER_TOKEN)) {
    Write-Host "RUNNER_TOKEN must be set"
    exit 1
}

if ((Test-Path env:RUNNER_ORG) -and (Test-Path env:RUNNER_REPO) -and (Test-Path env:RUNNER_ENTERPRISE)) {
    $ATTACH="$env:RUNNER_ORG/$env:RUNNER_REPO"
}elseif (Test-Path env:RUNNER_ORG) {
    $ATTACH="$env:RUNNER_ORG"
} elseif (Test-Path env:RUNNER_REPO) {
    $ATTACH="$env:RUNNER_REPO"
} elseif (Test-Path env:RUNNER_ENTERPRISE) {
    $ATTACH="enterprises/$env:RUNNER_ENTERPRISE"
} else {
    Write-Host "At least one of RUNNER_ORG or RUNNER_REPO or RUNNER_ENTERPRISE must be set"
    exit 1
}

if (-not (Test-Path env:RUNNER_REPO) -and (Test-Path env:RUNNER_GROUP)) {
    $RUNNER_GROUPS=$env:RUNNER_GROUP
}

$config_args = @()

if ((Test-Path env:RUNNER_EPHEMERAL)) {
    $config_args+="--ephemeral"
    Write-Host "Passing --ephemeral to config.sh to enable the ephemeral runner."
}

if ((Test-Path env:DISABLE_RUNNER_UPDATE)) {
    $config_args+="--disableupdate"
    Write-Host "Passing --disableupdate to config.sh to disable automatic runner updates."
}

$retries=10
while ($retries -ne 0) {
    Write-Host "Configuring github agent"
    .\config.cmd --unattended --replace `
        --url "$env:GITHUB_URL$ATTACH" `
        --token "$env:RUNNER_TOKEN" `
        --name "$env:RUNNER_NAME" `
        --labels "$env:RUNNER_LABELS" `
        --runnergroup "$env:RUNNER_GROUPS" `
        --work "$env:RUNNER_WORKDIR" $config_args

    if (Test-Path -Path .runner -PathType Leaf) {
        Write-Host "Runner successfully configured."
        break
    }
    Write-Host "Configuration failed. Retrying"
    $retries--
    sleep 1
}

if (-not (Test-Path -Path .runner -PathType Leaf)) {
    Write-Host "Configuration failed!"
    exit 1
}

Get-Content .runner

[isarkis]

@mazilu88 this is exactly what https://github.com/isarkis/actions-runner-controller/pull/1 is missing, can we add your change to that pull request?

[isarkis]

@antgrutta you wanted to combine efforts on adding Win support :) Take a look at https://github.com/isarkis/actions-runner-controller/pull/1, it's missing unit tests. Any help would be appreciated.

[mazilu88]

@mazilu88 this is exactly what https://github.com/isarkis/actions-runner-controller/pull/1 is missing, can we add your change to that pull request?

Sure

[GoranHalvarsson]

What is the status?

[bgrainger]

@GoranHalvarsson I don't think anything has been merged into this repo, but you can use the examples linked in the comments above to build your own Windows Container Image that executes the GitHub Actions runner. Here's the Dockerfile for the image I'm currently using: https://github.com/Faithlife/actions-runner-image/blob/master/windows/Dockerfile

(Getting your cluster set up to run and autoscale this image is something you need to manage, based on whether you're using EKS, AKS, GKE, etc.)