build icon indicating copy to clipboard operation
build copied to clipboard
verified publisher icon knative

Build reported as a timeout when it actually succeeded

Open scothis opened this issue 6 years ago • 0 comments

Expected Behavior

Successful builds completed before the timeout are reported as successful.

Actual Behavior

Successful builds can be reported as a timeout.

In this case, the build and pod are created at 2019-04-17T21:02:21Z, and finish 31 seconds later at 2019-04-17T21:02:52Z. Instead of the build's Succeeded condition reporting True the message is Build "square-build" failed to finish within "10m0s".

build

apiVersion: build.knative.dev/v1alpha1
kind: Build
metadata:
  creationTimestamp: "2019-04-17T21:02:21Z"
  generation: 1
  labels:
    projectriff.io/function: square
  name: square-build
  namespace: default
  resourceVersion: "2531684"
  selfLink: /apis/build.knative.dev/v1alpha1/namespaces/default/builds/square-build
  uid: 17a0e49f-6154-11e9-9a6b-42010a80016c
spec:
  Status: ""
  serviceAccountName: riff-build
  source:
    git:
      revision: master
      url: https://github.com/projectriff/fats.git
    subPath: functions/uppercase/node
  template:
    arguments:
    - name: IMAGE
      value: gcr.io/cf-sandbox-sandrews/square
    - name: CACHE
      value: cache
    kind: ClusterBuildTemplate
    name: riff-cnb
  timeout: 10m0s
  volumes:
  - name: cache
    persistentVolumeClaim:
      claimName: build-cache-square
status:
  builder: Cluster
  cluster:
    namespace: default
    podName: square-build-pod-4cad55
  completionTime: "2019-04-17T21:02:54Z"
  conditions:
  - lastTransitionTime: "2019-04-17T21:02:54Z"
    message: Build "square-build" failed to finish within "10m0s"
    reason: BuildTimeout
    status: "False"
    type: Succeeded
  startTime: "2019-04-17T20:52:54Z"
  stepsCompleted: null

pod (sorry there are so many steps)

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubernetes.io/limit-ranger: 'LimitRanger plugin set: cpu request for container
      nop; cpu request for init container build-step-credential-initializer; cpu request
      for init container build-step-git-source-0; cpu request for init container build-step-prepare;
      cpu request for init container build-step-detect; cpu request for init container
      build-step-analyze; cpu request for init container build-step-build; cpu request
      for init container build-step-export'
    sidecar.istio.io/inject: "false"
  creationTimestamp: "2019-04-17T21:02:21Z"
  labels:
    build.knative.dev/buildName: square-build
    projectriff.io/function: square
  name: square-build-pod-6070c4
  namespace: default
  ownerReferences:
  - apiVersion: build.knative.dev/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Build
    name: square-build
    uid: 17a0e49f-6154-11e9-9a6b-42010a80016c
  resourceVersion: "2531685"
  selfLink: /api/v1/namespaces/default/pods/square-build-pod-6070c4
  uid: 17b22b73-6154-11e9-9a6b-42010a80016c
spec:
  containers:
  - image: gcr.io/knative-releases/github.com/knative/build/cmd/nop@sha256:a318ee728d516ff732e2861c02ddf86197e52c6288049695781acb7710c841d4
    imagePullPolicy: IfNotPresent
    name: nop
    resources:
      requests:
        cpu: 100m
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: riff-build-token-5p282
      readOnly: true
  dnsPolicy: ClusterFirst
  initContainers:
  - args:
    - -basic-docker=push-credentials=https://gcr.io
    env:
    - name: HOME
      value: /builder/home
    image: gcr.io/knative-releases/github.com/knative/build/cmd/creds-init@sha256:ebf58f848c65c50a7158a155db7e0384c3430221564c4bbaf83e8fbde8f756fe
    imagePullPolicy: IfNotPresent
    name: build-step-credential-initializer
    resources:
      requests:
        cpu: 100m
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /workspace
      name: workspace
    - mountPath: /builder/home
      name: home
    - mountPath: /var/build-secrets/push-credentials
      name: secret-volume-push-credentials
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: riff-build-token-5p282
      readOnly: true
    workingDir: /workspace
  - args:
    - -url
    - https://github.com/projectriff/fats.git
    - -revision
    - master
    env:
    - name: HOME
      value: /builder/home
    image: gcr.io/knative-releases/github.com/knative/build/cmd/git-init@sha256:09f22919256ba4f7451e4e595227fb852b0a55e5e1e4694cb7df5ba0ad742b23
    imagePullPolicy: IfNotPresent
    name: build-step-git-source-0
    resources:
      requests:
        cpu: 100m
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /workspace
      name: workspace
    - mountPath: /builder/home
      name: home
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: riff-build-token-5p282
      readOnly: true
    workingDir: /workspace
  - args:
    - -c
    - |
      chown -R "1000:1000" "/builder/home" && chown -R "1000:1000" /layers && chown -R "1000:1000" /workspace
    command:
    - /bin/sh
    env:
    - name: HOME
      value: /builder/home
    image: ubuntu:18.04
    imagePullPolicy: Always
    name: build-step-prepare
    resources:
      requests:
        cpu: 100m
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /layers
      name: cache
    - mountPath: /workspace
      name: workspace
      subPath: functions/uppercase/node
    - mountPath: /builder/home
      name: home
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: riff-build-token-5p282
      readOnly: true
    workingDir: /workspace
  - args:
    - -app=/workspace
    - -group=/layers/group.toml
    - -plan=/layers/plan.toml
    command:
    - /lifecycle/detector
    env:
    - name: HOME
      value: /builder/home
    - name: RIFF
      value: "true"
    - name: RIFF_ARTIFACT
    - name: RIFF_HANDLER
    - name: RIFF_OVERRIDE
    image: projectriff/builder:0.2.0
    imagePullPolicy: Always
    name: build-step-detect
    resources:
      requests:
        cpu: 100m
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /layers
      name: cache
    - mountPath: /workspace
      name: workspace
      subPath: functions/uppercase/node
    - mountPath: /builder/home
      name: home
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: riff-build-token-5p282
      readOnly: true
    workingDir: /workspace
  - args:
    - -layers=/layers
    - -helpers=true
    - -group=/layers/group.toml
    - gcr.io/cf-sandbox-sandrews/square
    command:
    - /lifecycle/analyzer
    env:
    - name: HOME
      value: /builder/home
    image: projectriff/builder:0.2.0
    imagePullPolicy: Always
    name: build-step-analyze
    resources:
      requests:
        cpu: 100m
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /layers
      name: cache
    - mountPath: /workspace
      name: workspace
      subPath: functions/uppercase/node
    - mountPath: /builder/home
      name: home
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: riff-build-token-5p282
      readOnly: true
    workingDir: /workspace
  - args:
    - -layers=/layers
    - -app=/workspace
    - -group=/layers/group.toml
    - -plan=/layers/plan.toml
    command:
    - /lifecycle/builder
    env:
    - name: HOME
      value: /builder/home
    - name: RIFF
      value: "true"
    - name: RIFF_ARTIFACT
    - name: RIFF_HANDLER
    - name: RIFF_OVERRIDE
    image: projectriff/builder:0.2.0
    imagePullPolicy: Always
    name: build-step-build
    resources:
      requests:
        cpu: 100m
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /layers
      name: cache
    - mountPath: /workspace
      name: workspace
      subPath: functions/uppercase/node
    - mountPath: /builder/home
      name: home
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: riff-build-token-5p282
      readOnly: true
    workingDir: /workspace
  - args:
    - -layers=/layers
    - -helpers=true
    - -app=/workspace
    - -image=packs/run:0.1.0
    - -group=/layers/group.toml
    - gcr.io/cf-sandbox-sandrews/square
    command:
    - /lifecycle/exporter
    env:
    - name: HOME
      value: /builder/home
    image: projectriff/builder:0.2.0
    imagePullPolicy: Always
    name: build-step-export
    resources:
      requests:
        cpu: 100m
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /layers
      name: cache
    - mountPath: /workspace
      name: workspace
      subPath: functions/uppercase/node
    - mountPath: /builder/home
      name: home
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: riff-build-token-5p282
      readOnly: true
    workingDir: /workspace
  nodeName: gke-riff-default-pool-f902d6b2-qps0
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: riff-build
  serviceAccountName: riff-build
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: cache
    persistentVolumeClaim:
      claimName: build-cache-square
  - emptyDir: {}
    name: empty-dir
  - emptyDir: {}
    name: workspace
  - emptyDir: {}
    name: home
  - name: secret-volume-push-credentials
    secret:
      defaultMode: 420
      secretName: push-credentials
  - name: riff-build-token-5p282
    secret:
      defaultMode: 420
      secretName: riff-build-token-5p282
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2019-04-17T21:02:53Z"
    reason: PodCompleted
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2019-04-17T21:02:23Z"
    reason: PodCompleted
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2019-04-17T21:02:23Z"
    reason: PodCompleted
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2019-04-17T21:02:23Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://8bc79cc6b1c2f06ab84de9dd379f463d51b0310840c9f55ca0413a146d038e44
    image: sha256:94cec99fc35341544691c5501ba27d0ed7b659a1e7026c9b8cd2f6d6c7dfb310
    imageID: docker-pullable://gcr.io/knative-releases/github.com/knative/build/cmd/nop@sha256:a318ee728d516ff732e2861c02ddf86197e52c6288049695781acb7710c841d4
    lastState: {}
    name: nop
    ready: false
    restartCount: 0
    state:
      terminated:
        containerID: docker://8bc79cc6b1c2f06ab84de9dd379f463d51b0310840c9f55ca0413a146d038e44
        exitCode: 0
        finishedAt: "2019-04-17T21:02:53Z"
        reason: Completed
        startedAt: "2019-04-17T21:02:53Z"
  hostIP: 10.128.0.12
  initContainerStatuses:
  - containerID: docker://793a3b89ec971a7f2edc44e01d27c137e56f5a03026631114554a3689fdc3264
    image: sha256:ea5d3f4e41798996e3962d6f2398f508e83db313b14574128e5ec0699abbcbc8
    imageID: docker-pullable://gcr.io/knative-releases/github.com/knative/build/cmd/creds-init@sha256:ebf58f848c65c50a7158a155db7e0384c3430221564c4bbaf83e8fbde8f756fe
    lastState: {}
    name: build-step-credential-initializer
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: docker://793a3b89ec971a7f2edc44e01d27c137e56f5a03026631114554a3689fdc3264
        exitCode: 0
        finishedAt: "2019-04-17T21:02:34Z"
        reason: Completed
        startedAt: "2019-04-17T21:02:34Z"
  - containerID: docker://b66538c49e9f0358e2aec006f5678d7cb295141276e74658ea3ad890ae8e3c2b
    image: sha256:97f4e092c75a3f15e258b74b3c1b9fb9cb5fb3bb087a58da0452cd3eb3dea2a6
    imageID: docker-pullable://gcr.io/knative-releases/github.com/knative/build/cmd/git-init@sha256:09f22919256ba4f7451e4e595227fb852b0a55e5e1e4694cb7df5ba0ad742b23
    lastState: {}
    name: build-step-git-source-0
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: docker://b66538c49e9f0358e2aec006f5678d7cb295141276e74658ea3ad890ae8e3c2b
        exitCode: 0
        finishedAt: "2019-04-17T21:02:35Z"
        reason: Completed
        startedAt: "2019-04-17T21:02:34Z"
  - containerID: docker://7fe03c572cc965207c6ee0ecd915fb0924d17f0d670a8d65135c0ec5d83a1205
    image: ubuntu:18.04
    imageID: docker-pullable://ubuntu@sha256:f2557f94cac1cc4509d0483cb6e302da841ecd6f82eb2e91dc7ba6cfd0c580ab
    lastState: {}
    name: build-step-prepare
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: docker://7fe03c572cc965207c6ee0ecd915fb0924d17f0d670a8d65135c0ec5d83a1205
        exitCode: 0
        finishedAt: "2019-04-17T21:02:35Z"
        reason: Completed
        startedAt: "2019-04-17T21:02:35Z"
  - containerID: docker://52d54ef79aeb4d294d5470f6983b4f25a2e18702fb45346b9f2619efed595b18
    image: projectriff/builder:0.2.0
    imageID: docker-pullable://projectriff/builder@sha256:dccac174b2e80186aab20ed3a52a61e8a69d6c023e16f45c53c6385d7f5b5c4c
    lastState: {}
    name: build-step-detect
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: docker://52d54ef79aeb4d294d5470f6983b4f25a2e18702fb45346b9f2619efed595b18
        exitCode: 0
        finishedAt: "2019-04-17T21:02:37Z"
        reason: Completed
        startedAt: "2019-04-17T21:02:37Z"
  - containerID: docker://a98ce05337738a7c633acf2ca65c9902542359ac038e745af0dcef8e1d812c5b
    image: projectriff/builder:0.2.0
    imageID: docker-pullable://projectriff/builder@sha256:dccac174b2e80186aab20ed3a52a61e8a69d6c023e16f45c53c6385d7f5b5c4c
    lastState: {}
    name: build-step-analyze
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: docker://a98ce05337738a7c633acf2ca65c9902542359ac038e745af0dcef8e1d812c5b
        exitCode: 0
        finishedAt: "2019-04-17T21:02:38Z"
        reason: Completed
        startedAt: "2019-04-17T21:02:38Z"
  - containerID: docker://3bc87df69cb5166f261fcaa6a3f8ec7bbb7504da2e62066491c535bdabb53b05
    image: projectriff/builder:0.2.0
    imageID: docker-pullable://projectriff/builder@sha256:dccac174b2e80186aab20ed3a52a61e8a69d6c023e16f45c53c6385d7f5b5c4c
    lastState: {}
    name: build-step-build
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: docker://3bc87df69cb5166f261fcaa6a3f8ec7bbb7504da2e62066491c535bdabb53b05
        exitCode: 0
        finishedAt: "2019-04-17T21:02:46Z"
        reason: Completed
        startedAt: "2019-04-17T21:02:40Z"
  - containerID: docker://73aecaac4b79a7438e079b6b4d90e7585aadc31521503e6a8731f6443c248e26
    image: projectriff/builder:0.2.0
    imageID: docker-pullable://projectriff/builder@sha256:dccac174b2e80186aab20ed3a52a61e8a69d6c023e16f45c53c6385d7f5b5c4c
    lastState: {}
    name: build-step-export
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: docker://73aecaac4b79a7438e079b6b4d90e7585aadc31521503e6a8731f6443c248e26
        exitCode: 0
        finishedAt: "2019-04-17T21:02:52Z"
        reason: Completed
        startedAt: "2019-04-17T21:02:47Z"
  phase: Succeeded
  podIP: 10.48.0.92
  qosClass: Burstable
  startTime: "2019-04-17T21:02:23Z"

Steps to Reproduce the Problem

  1. create a build
  2. ...?
  3. see that the build is reported as a timeout, when it actually succeeded

I don't have steps to reproduce at this moment, but wanted to capture the issue in case others see similar behavior.

Additional Info

Knative build 0.5.0, GKE v1.12.6-gke.10

scothis avatar Apr 17 '19 21:04 scothis