ephemeralStorage changes ephemeral-storage for init container but not kaniko container

[jlewi]

Expected behavior

Setting ephmeralStorage when using the kaniko builder should set ephmeral-storage for the kaniko container.

Actual behavior

I observe ephmeral-storage being changed for the init container but not the kaniko container.

This is illustrated in the pod spec below. The skafold.yaml sets "ephmeralStorage = 30Gi". Init container has 30GI of ephmeral storage but the kaniko container only has 10GI.

apiVersion: v1
kind: Pod
metadata:
  annotations:
    autopilot.gke.io/resource-adjustment: '{"input":{"initContainers":[{"requests":{"cpu":"4","ephemeral-storage":"30Gi","memory":"8Gi"},"name":"kaniko-init-container"}],"containers":[{"requests":{"cpu":"4","ephemeral-storage":"30Gi","memory":"8Gi"},"name":"kaniko"}]},"output":{"initContainers":[{"limits":{"cpu":"4","ephemeral-storage":"30Gi","memory":"8Gi"},"requests":{"cpu":"4","ephemeral-storage":"30Gi","memory":"8Gi"},"name":"kaniko-init-container"}],"containers":[{"limits":{"cpu":"4","ephemeral-storage":"10Gi","memory":"8Gi"},"requests":{"cpu":"4","ephemeral-storage":"10Gi","memory":"8Gi"},"name":"kaniko"}]},"modified":true}'
    seccomp.security.alpha.kubernetes.io/pod: runtime/default
  creationTimestamp: "2022-07-28T21:25:46Z"
  generateName: kaniko-
  labels:
    skaffold-kaniko: skaffold-kaniko
  name: kaniko-dbsnw
  namespace: kaniko
  resourceVersion: "2348302"
  uid: 81cb7911-ab0d-47de-88b3-9112a4d09311
spec:
  containers:
  - args:
    - --destination
    - gcr.io/dev-bytetoko/devbox:e7d6f67-dirty
    - --dockerfile
    - Dockerfile
    - --context
    - dir:///kaniko/buildcontext
    env:
    - name: UPSTREAM_CLIENT_TYPE
      value: UpstreamClient(skaffold-v1.38.0)
    - name: IMAGE_TAG
      value: e7d6f67-dirty
    - name: IMAGE_REPO
      value: gcr.io/dev-bytetoko
    - name: IMAGE_NAME
      value: devbox
    image: gcr.io/kaniko-project/executor:latest
    imagePullPolicy: IfNotPresent
    name: kaniko
    resources:
      limits:
        cpu: "4"
        ephemeral-storage: 10Gi
        memory: 8Gi
      requests:
        cpu: "4"
        ephemeral-storage: 10Gi
        memory: 8Gi
    securityContext:
      capabilities:
        drop:
        - NET_RAW
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /kaniko/buildcontext
      name: kaniko-emptydir
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-gm58j
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  initContainers:
  - command:
    - sh
    - -c
    - while [ ! -f /tmp/complete ]; do sleep 1; done
    image: gcr.io/k8s-skaffold/skaffold-helpers/busybox
    imagePullPolicy: IfNotPresent
    name: kaniko-init-container
    resources:
      limits:
        cpu: "4"
        ephemeral-storage: 30Gi
        memory: 8Gi
      requests:
        cpu: "4"
        ephemeral-storage: 30Gi
        memory: 8Gi
    securityContext:
      capabilities:
        drop:
        - NET_RAW
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /kaniko/buildcontext
      name: kaniko-emptydir
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-gm58j
      readOnly: true
  nodeName: gk3-dev-nap-n5lv5dg1-335cd0ca-cvnd
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Never
  schedulerName: gke.io/optimize-utilization-scheduler
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  serviceAccount: kaniko
  serviceAccountName: kaniko
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - emptyDir: {}
    name: kaniko-emptydir
  - name: kube-api-access-gm58j
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2022-07-28T21:28:01Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2022-07-28T21:28:03Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2022-07-28T21:28:03Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2022-07-28T21:27:52Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: containerd://7ded284fe8512005b93ea677510de3d0bcddd65801c10e13775bdcebd1b80c44
    image: gcr.io/kaniko-project/executor:latest
    imageID: gcr.io/kaniko-project/executor@sha256:b44b0744b450e731b5a5213058792cd8d3a6a14c119cf6b1f143704f22a7c650
    lastState: {}
    name: kaniko
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2022-07-28T21:28:03Z"
  hostIP: 10.138.0.30
  initContainerStatuses:
  - containerID: containerd://04e3a23e5e69da3ba43b592d82c7edfb5812f6e00ab0155a8496ff94c9b16a06
    image: gcr.io/k8s-skaffold/skaffold-helpers/busybox:latest
    imageID: gcr.io/k8s-skaffold/skaffold-helpers/busybox@sha256:ee44b399df993016003bf5466bd3eeb221305e9d0fa831606bc7902d149c775b
    lastState: {}
    name: kaniko-init-container
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: containerd://04e3a23e5e69da3ba43b592d82c7edfb5812f6e00ab0155a8496ff94c9b16a06
        exitCode: 0
        finishedAt: "2022-07-28T21:27:58Z"
        reason: Completed
        startedAt: "2022-07-28T21:27:55Z"
  phase: Running
  podIP: 10.107.128.67
  podIPs:
  - ip: 10.107.128.67
  qosClass: Guaranteed
  startTime: "2022-07-28T21:27:52Z"

Information

• Skaffold version: v1.38.0
• Operating system: System Version: macOS 12.2.1 (21D62) Kernel Version: Darwin 21.3.0
• Installed via: gcloud
• Contents of skaffold.yaml:

apiVersion: skaffold/v2beta28
kind: Config
metadata:
  name: vscode
build:
  artifacts:
  - image: gcr.io/dev-bytetoko/devbox
    context: .
    kaniko:
      dockerfile: Dockerfile
  # googleCloudBuild:
  #   projectId: dev-bytetoko
  #   machineType: E2_HIGHCPU_8
  cluster:
    namespace: kaniko
    serviceAccount: kaniko
    resources:
      requests:
        cpu: 4
        memory: 8Gi
        ephemeralStorage: 30Gi
      limits:
        cpu: 4
        memory: 8Gi
        ephemeralStorage: 30Gi
deploy:
  kustomize: {}

Steps to reproduce the behavior

1. Run skaffold build with the above skaffold.yaml file and any Dockerfile
2. Observe the kaniko pod created doesn't have ephmeralStorage set as desired

Looks like #2990 is the issue that added support for ephmeralStorage

[gsquared94]

From the docs for Autopilot GKE cluster:

The ephemeral storage limit, which must be within 10 MiB and 10 GiB for all compute classes.

So the ephemeralStorage is being modified by the Autopilot GKE cluster (see the autopilot.gke.io/resource-adjustment annotation). Closing this issue, as it's not related to Skaffold.

[jlewi]

@gsquared94 Thank you very much.