kuttl icon indicating copy to clipboard operation
kuttl copied to clipboard
verified publisher icon kudobuilder

Unable to assert success on attributes of pod other than phase

Open samarth-gupta-traceable opened this issue 5 years ago • 7 comments

Edit 2025-02-03:

Note the recently added CEL support which should help with this issue

I am trying below:

  1. install a helm chart via command provided in TestSuite in file kuttl-test.yaml 2.assert if the pod started successfully as part of helm install, via 00-assert.yaml which is inside a test dir. Success criteria is: (1. Pod in Running state, 2. Pod should have zero restarts, 3. Ready probe should be true)

I am having trouble with step 2 i.e. assert: I have tried both below as content of assert.yaml but both of them result in test fail, when actually Pod has come up and in Running state.

below 2 config fails

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: gateway-service
status:
  containerStatuses:
    name: gateway-service
    ready: true

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: gateway-service
status:
  phase: Running
  conditions:
    status: "True"
    type: Ready

below passes

apiVersion: v1
kind: Pod
metadata:
  labels:
    app: gateway-service
status:
  phase: Running

below is the error I get in failures

=== CONT  kuttl/harness/service-bootstrap
    kuttl/harness/service-bootstrap: logger.go:42: 11:07:05 | service-bootstrap | Skipping creation of user-supplied namespace: testnamespace
    kuttl/harness/service-bootstrap: logger.go:42: 11:07:05 | service-bootstrap/0- | starting test step 0-
    kuttl/harness/service-bootstrap: logger.go:42: 11:08:06 | service-bootstrap/0- | test step failed 0-
    kuttl/harness/service-bootstrap: logger.go:42: 11:08:06 | service-bootstrap | Skipping deletion of user-supplied namespace: testnamespace
--- FAIL: kuttl (62.91s)
    --- FAIL: kuttl/harness (60.96s)
        --- FAIL: kuttl/harness/service-bootstrap (60.96s)
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x2028e1d]
goroutine 288 [running]:
testing.tRunner.func1.1(0x2166740, 0x3151610)
	/usr/local/Cellar/go/1.14.4/libexec/src/testing/testing.go:940 +0x2f5
testing.tRunner.func1(0xc0002070e0)
	/usr/local/Cellar/go/1.14.4/libexec/src/testing/testing.go:943 +0x3f9
panic(0x2166740, 0x3151610)
	/usr/local/Cellar/go/1.14.4/libexec/src/runtime/panic.go:969 +0x166
github.com/kudobuilder/kuttl/pkg/test.(*Step).Run(0xc0004dc630, 0xc000398a50, 0x9, 0xc000554920, 0xc0002070e0, 0xc0002e6338)
	/Users/kensipe/projects/go/src/github.com/kudobuilder/kuttl/pkg/test/step.go:413 +0x72d

below is the kubectl get pod output

apiVersion: v1
kind: Pod
metadata:
  annotations:
    checksum/config: 56ddbd66a930783027919a5a50d4097c99d4
  creationTimestamp: "2020-07-02T05:26:05Z"
  generateName: gateway-service-55b678f956-
  labels:
    app: gateway-service
    pod-template-hash: 55b678f956
    release: gateway-service
  name: gateway-service-55b678f956-cm5tt
  namespace: testnamespace
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: gateway-service-55b678f956
    uid: d743dc21-08ad-4b8e-b2b0-c0aed297a295
  resourceVersion: "3382861"
  selfLink: /api/v1/namespaces/testnamespace/pods/gateway-service-55b678f956-cm5tt
  uid: 878717f8-a7ad-46d4-83ca-8079e366880a
spec:
  containers:
  - env:
    - name: SERVICE_NAME
      value: gateway-service
    - name: POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: BOOTSTRAP_CONFIG_URI
      value: file:///app/resources/configs
    - name: LOG4J_CONFIGURATION_FILE
      value: /app/log/log4j2.properties
    - name: JAVA_TOOL_OPTIONS
      value: -XX:InitialRAMPercentage=50.0 -XX:MaxRAMPercentage=75.0
    - name: CLUSTER_NAME
      value: staging
    image: test-docker.jfrog.io/ai.test.platform/gateway-service:0.1.77
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 3
      initialDelaySeconds: 10
      periodSeconds: 10
      successThreshold: 1
      tcpSocket:
        port: http
      timeoutSeconds: 1
    name: gateway-service
    ports:
    - containerPort: 50071
      name: http
      protocol: TCP
    - containerPort: 50072
      name: admin-port
      protocol: TCP
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: /health
        port: admin-port
        scheme: HTTP
      initialDelaySeconds: 2
      periodSeconds: 5
      successThreshold: 1
      timeoutSeconds: 1
    resources:
      limits:
        cpu: 200m
        memory: 1Gi
      requests:
        cpu: 100m
        memory: 512Mi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /app/log
      name: log4j-config
    - mountPath: /app/resources/configs/gateway-service/staging/application.conf
      name: service-config
      subPath: application.conf
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: default-token-qz24h
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  imagePullSecrets:
  - name: regcred
  nodeName: docker-desktop
  nodeSelector:
    workload-general: enabled
  priority: 0
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - configMap:
      defaultMode: 420
      name: gateway-service-config
    name: service-config
  - configMap:
      defaultMode: 420
      name: gateway-kafka-log-appender-config
    name: log4j-config
  - name: default-token-qz24h
    secret:
      defaultMode: 420
      secretName: default-token-qz24h
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2020-07-02T05:26:05Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2020-07-02T05:26:35Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2020-07-02T05:26:35Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2020-07-02T05:26:05Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - containerID: docker://3addd4a490569630114480955f6c75c38ee177597f577598ac73f9d2c3a7245a
    image: test-docker.jfrog.io/ai.test.platform/gateway-service:0.1.77
    imageID: docker://sha256:922a76566b44d5243810598fa8e2a28245e65f14d4093fec1f521e213535aad2
    lastState: {}
    name: gateway-service
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2020-07-02T05:26:06Z"
  hostIP: 192.168.65.3
  phase: Running
  podIP: 10.1.2.247
  podIPs:
  - ip: 10.1.2.247
  qosClass: Burstable
  startTime: "2020-07-02T05:26:05Z"

Another doubt I had was in the examples i see pod phase mentioned as Successful: https://kuttl.dev/docs/testing/asserts-errors.html#format

status:
  phase: Successful

however valid phases mentioned as pod lifecycle are: Pending , Running , Succeeded, Failed here: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-phase

samarth-gupta-traceable avatar Jul 02 '20 06:07 samarth-gupta-traceable

hey @samarth-gupta-traceable can you please tell us the version of KUTTL you are using and also the directory structure as the error shouldn't be a panic.

And also for your case you want to assert containerStatuses which is an array with

conditions:
    status: "True"
    type: Ready

which isn't an array. Same for the containerStatuses we have an issue open for array support https://github.com/kudobuilder/kuttl/issues/76

zmalik avatar Jul 02 '20 07:07 zmalik

@zmalik
KUTTL version: 0.5.0

Dir structure:

kuttl-test.yaml
kuttl-test
    - service-bootstrap
            - 00-assert.yaml
            - 01-cleanup.yaml

also, meanwhile is there a workaround to assert on array , or is there a way to assert of liveness or Readiness probe events by kubenertes ?

thanks

samarth-gupta-traceable avatar Jul 02 '20 07:07 samarth-gupta-traceable

@samarth-gupta-traceable here is one example https://github.com/kudobuilder/kuttl/blob/main/pkg/test/test_data/crd-in-step/00-assert.yaml the order matter in this case

zmalik avatar Jul 02 '20 09:07 zmalik

yeah @samarth-gupta-traceable great comment regarding the use of "successful" in a phase in our docs :(. that is no good. I'll fix that today

kensipe avatar Jul 02 '20 12:07 kensipe

repeated from slack so it isn't lost

@samarth thank you for report on this... there are a number of issues:

  1. poor choice in phase values in docs
  2. asserting on an array
  3. segfault

I will be review this more closely today... but it seems like you are in good hands with Zain actions:

  1. docs will be fixed today
  2. segfault will be fixed today
  3. We will see if we can take a look at arrays... I was expecting that a more expressive language may be necessary here... but we will see

kensipe avatar Jul 02 '20 12:07 kensipe

I think i'm also hitting the same segfault issue, i've created #154 with my scenario.

luciano-nbs avatar Jul 02 '20 19:07 luciano-nbs