[Feature] Tutorial on Deploying Headscale with TalosOS

[8-cm]

Use case

A comprehensive tutorial on deploying Headscale on TalosOS would greatly assist users in setting up their own self-hosted VPN solutions using the secure and modern infrastructure provided by TalosOS. This will benefit users looking for robust, scalable, and secure networking solutions.

SideroLabs already has a great extension for Tailscale client side, so having own control server node is just natural.

Description

The community would greatly benefit from a step-by-step tutorial focused on deploying Headscale, an open-source Tailscale control server, on TalosOS. This guide should detail the prerequisites, installation steps, and configuration, along with operational advice for maintaining a Headscale server. Given that SideroLabs already offers a well-regarded extension for the Tailscale client-side, providing a tutorial on setting up a control server on TalosOS complements this existing resource and fulfills a natural progression for users aiming to leverage a complete Tailscale solution in their environments.

Contribution

• [ ] I can write the design doc for this feature
• [ ] I can contribute this feature

How can it be implemented?

Description: I am currently facing an issue where the main container for Headscale, deployed using Helm on Kubernetes, exits immediately after starting, with the container returning an exit code of 1. There are no logs being generated that indicate the cause of the failure, which makes troubleshooting challenging.

Steps to Reproduce:

1. Deploy Headscale using the provided Helm chart - gabe565/headscale.
2. Monitor the pod status with kubectl get pods -w and kubectl logs <pod-name>.
3. Observe that the Headscale pod enters a CrashLoopBackOff state.

Expected Behavior: The Headscale pod should start successfully and remain in a running state without crashing.

Actual Behavior: The Headscale pod crashes immediately after starting and continues to crash in a loop with exit code 1.

Environment:

• Kubernetes version: v1.30.0
• Cloud provider or hardware configuration: Proxmox VM
• OS: TalosOS 1.8.0
• Install tools: Helm template | kubectl apply -f-
• Others:

Additional Information:

• The Helm chart used is version headscale-0.13.1 with Headscale version 0.22.3.
• Persistent volumes and services appear to deploy without issue.
• None (really not a single line of log - just exit 1) logs are available from the Headscale pod, even when attempting to fetch logs using kubectl logs <pod_name> or -p logs.

Request for Help: Could anyone provide insights or suggestions on what might be causing this immediate crash or how to enable more detailed logging to diagnose the issue further? Any help or pointers towards additional debugging steps would be greatly appreciated.

---
# Source: headscale/templates/common.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: headscale-config
  labels:
    app.kubernetes.io/instance: headscale
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: headscale
    app.kubernetes.io/version: 0.22.3
    helm.sh/chart: headscale-0.13.1
  annotations:
    "helm.sh/resource-policy": keep
spec:
  accessModes:
    - "ReadWriteMany"
  resources:
    requests:
      storage: "5Gi"
  storageClassName: "example"
---
# Source: headscale/templates/common.yaml
apiVersion: v1
kind: Service
metadata:
  name: headscale
  labels:
    app.kubernetes.io/service: headscale
    app.kubernetes.io/instance: headscale
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: headscale
    app.kubernetes.io/version: 0.22.3
    helm.sh/chart: headscale-0.13.1
spec:
  type: ClusterIP
  ports:
    - port: 8080
      targetPort: http
      protocol: TCP
      name: http
    - port: 9090
      targetPort: metrics
      protocol: TCP
      name: metrics
  selector:
    app.kubernetes.io/instance: headscale
    app.kubernetes.io/name: headscale
---
# Source: headscale/templates/common.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: headscale
  labels:
    app.kubernetes.io/instance: headscale
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: headscale
    app.kubernetes.io/version: 0.22.3
    helm.sh/chart: headscale-0.13.1
spec:
  revisionHistoryLimit: 3
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app.kubernetes.io/name: headscale
      app.kubernetes.io/instance: headscale
  template:
    metadata:
      labels:
        app.kubernetes.io/name: headscale
        app.kubernetes.io/instance: headscale
    spec:
      serviceAccountName: default
      automountServiceAccountToken: true
      dnsPolicy: ClusterFirst
      enableServiceLinks: true
      initContainers:
        - command:
          - sh
          - -c
          - |
            if [[ -f "$CONFIG_DIR/config.yaml" ]]; then
              echo 'Config already exists' >&2
            else
              echo 'Writing empty config' >&2
              mkdir -p "$CONFIG_DIR"
              cat <<- 'EOF' >"$CONFIG_DIR/config.yaml"
                # It's suggested to use environment variables to configure Headscale.
                # For config reference, see https://github.com/juanfont/headscale/blob/main/config-example.yaml
                # To configure any of these as an env:
                #   1. Flatten object keys using "_"
                #   2. Prefix with "HEADSCALE_"
                #
                # For example:
                #   - "listen_addr" becomes "HEADSCALE_LISTEN_ADDR"
                #   - "log.level" becomes "HEADSCALE_LOG_LEVEL"
            EOF
            fi
          env:
          - name: CONFIG_DIR
            value: /etc/headscale
          image: alpine
          name: config
          volumeMounts:
          - mountPath: /etc/headscale
            name: config
      containers:
        - name: headscale
          image: ghcr.io/juanfont/headscale:0.22.3
          imagePullPolicy: IfNotPresent
          args:
            - serve
          env:
            - name: HEADSCALE_DB_PATH
              value: /etc/headscale/db.sqlite
            - name: HEADSCALE_DB_TYPE
              value: sqlite3
            - name: HEADSCALE_DERP_AUTO_UPDATE_ENABLED
              value: "true"
            - name: HEADSCALE_DERP_UPDATE_FREQUENCY
              value: 24h
            - name: HEADSCALE_DERP_URLS
              value: https://controlplane.tailscale.com/derpmap/default
            - name: HEADSCALE_DNS_CONFIG_BASE_DOMAIN
              value: example.com
            - name: HEADSCALE_DNS_CONFIG_MAGIC_DNS
              value: "true"
            - name: HEADSCALE_DNS_CONFIG_NAMESERVERS
              value: 1.1.1.1 1.0.0.1
            - name: HEADSCALE_EPHEMERAL_NODE_INACTIVITY_TIMEOUT
              value: 30m
            - name: HEADSCALE_IP_PREFIXES
              value: "fd7a:115c:a1e0::/48 100.64.0.0/10"
            - name: HEADSCALE_LISTEN_ADDR
              value: 0.0.0.0:8080
            - name: HEADSCALE_METRICS_LISTEN_ADDR
              value: 0.0.0.0:9090
            - name: HEADSCALE_NOISE
              value: '{}'
            - name: HEADSCALE_NOISE_PRIVATE_KEY_PATH
              value: /etc/headscale/noise_private.key
            - name: HEADSCALE_PRIVATE_KEY_PATH
              value: /etc/headscale/private.key
            - name: HEADSCALE_SERVER_URL
              value: https://example.com
          ports:
            - name: http
              containerPort: 8080
              protocol: TCP
            - name: metrics
              containerPort: 9090
              protocol: TCP
            - name: ui
              containerPort: 80
              protocol: TCP
          volumeMounts:
            - name: config
              mountPath: /etc/headscale
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 0
            periodSeconds: 10
            tcpSocket:
              port: 8080
            timeoutSeconds: 1
          readinessProbe:
            failureThreshold: 3
            initialDelaySeconds: 0
            periodSeconds: 10
            tcpSocket:
              port: 8080
            timeoutSeconds: 1
          startupProbe:
            failureThreshold: 30
            initialDelaySeconds: 0
            periodSeconds: 5
            tcpSocket:
              port: 8080
            timeoutSeconds: 1
      volumes:
        - name: config
          persistentVolumeClaim:
            claimName: headscale-config
---
# Source: headscale/templates/common.yaml
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
  name: headscale
  labels:
    app.kubernetes.io/instance: headscale
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: headscale
    app.kubernetes.io/version: 0.22.3
    helm.sh/chart: headscale-0.13.1
spec:
  tls:
    - hosts:
        - "tls-example-com"
  rules:
    - host: "example.com"
      http:
        paths:
          - path: "/"
            pathType: Prefix
            backend:
              service:
                name: headscale
                port:
                  number: 8080
---
# Source: headscale/templates/common.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: headscale
  labels:
    app.kubernetes.io/instance: headscale
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: headscale
    app.kubernetes.io/version: 0.22.3
    helm.sh/chart: headscale-0.13.1
spec:
  selector:
    matchLabels:
      app.kubernetes.io/service: headscale
      app.kubernetes.io/name: headscale
      app.kubernetes.io/instance: headscale
  endpoints:
    - interval: 30s
      path: /metrics
      port: metrics
      scheme: http
      scrapeTimeout: 10s