HTTP Cache Kubernetes Server Service

[saienduri]

Request description

We want to introduce an http cache server to the kubernetes cluster as it will help with the build times for several jobs:

linux_x64_clang in ci_linux_x64_clang.yml linux_x64_clang_asan in ci_linux_x64_clang_asan.yml linux_x64_clang_tsan in ci_linux_x64_clang_tsan.yml linux_x64_clang_debug in ci_linux_x64_clang_debug.yml ci_linux_x64_bazel in ci_linux_x64_bazel.yml

To configure an http cache server as a service for a Kubernetes cluster, you can follow these steps. This is just an initial draft of how the overall structure would look like, but might need changes at each step to get it working properly:

1. Create a Kubernetes Deployment for the HTTP Nginx Cache Server

Create a deployment that will run the HTTP server as a pod. Here is maybe something that the deployment YAML would like:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: http-cache-server
spec:
  replicas: 1
  selector:
    matchLabels:
      app: http-cache
  template:
    metadata:
      labels:
        app: http-cache
    spec:
      containers:
        - name: nginx
          image: nginx:latest
          volumeMounts:
            - name: cache-volume
              mountPath: /usr/share/nginx/html/cache
          ports:
            - containerPort: 80
      volumes:
        - name: cache-volume
          emptyDir: {}

kubectl apply -f nginx-deployment.yaml

3. Create Kubernetes Service for the sccache server

This is so that the http cache server is exposed to our cluster, and the runners can actually access it. The service yaml file should look something like this:

apiVersion: v1
kind: Service
metadata:
  name: http-cache-server
spec:
  selector:
    app: http-cache
  ports:
    - protocol: TCP
      port: 4226
      targetPort: 4226
  type: ClusterIP

kubectl apply -f nginx-service.yaml

4. Github Runner Configuration (sccache, will have to do something similar for bazel)

On the runner machines, configure sccache to point to the server by setting the SCCACHE_SERVER_URL environment variable and set the size:

SCCACHE_SERVER_URL=http://<http-cache-server>:4226
SCCACHE_CACHE_SIZE=10G

I think that this is just adding this block here: https://github.com/saienduri/AKS-GitHubARC-Setup/blob/main/latest-config-files/values.yaml#L28

Replace with the internal service name (kubectl get services -n <namespace> to get the name).

This should be all that's needed to get started, but we will have to figure out the anonymous read only behavior

What component(s) does this issue relate to?

Other

Additional context

This feature will help bring down our builder job times by a significant amount.