capsule-render icon indicating copy to clipboard operation
capsule-render copied to clipboard

Sonobuoy results may count incorrectly in some cases

Open johnSchnake opened this issue 2 years ago • 0 comments

What steps did you take and what happened: Ran e2e tests in parallel with the postprocessor to remove skips. Final result had some xml files with no tests. So the 'k8s test suite' was the leaf node (which is unusual). When counting tests, sonobuoy then reported 9 tests instead of the 5 I expected.

File looks like:

name: e2e-0
status: unknown
meta:
  type: summary
items:
- name: sonobuoy_results.yaml
  status: ""
  meta:
    file: results/global/sonobuoy_results.yaml
    type: file
  items:
  - name: junit_01.xml
    status: passed
    meta:
      file: /tmp/sonobuoy/results/junit_01.xml
      type: file
    items:
    - name: Kubernetes e2e suite
      status: passed
      items:
      - name: '[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] patching/updating
          a mutating webhook should work [Conformance]'
        status: passed
      - name: '[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] patching/updating
          a validating webhook should work [Conformance]'
        status: passed
      - name: '[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] listing
          mutating webhooks should work [Conformance]'
        status: passed
  - name: junit_02.xml
    status: passed
    meta:
      file: /tmp/sonobuoy/results/junit_02.xml
      type: file
    items:
    - name: Kubernetes e2e suite
      status: passed
      items:
      - name: '[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] listing
          validating webhooks should work [Conformance]'
        status: passed
  - name: junit_03.xml
    status: passed
    meta:
      file: /tmp/sonobuoy/results/junit_03.xml
      type: file
    items:
    - name: Kubernetes e2e suite
      status: passed
      items:
      - name: '[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should
          be able to deny attaching pod [Conformance]'
        status: passed
  - name: junit_04.xml
    status: passed
    meta:
      file: /tmp/sonobuoy/results/junit_04.xml
      type: file
    items:
    - name: Kubernetes e2e suite
      status: passed
  - name: junit_05.xml
    status: passed
    meta:
      file: /tmp/sonobuoy/results/junit_05.xml
      type: file
    items:
    - name: Kubernetes e2e suite
      status: passed
...

Part of the problem is in the post-processor plugin; it removed a bunch of tests and left none of the original leaves. However, the junit processing should think about how to handle the 'suite' in general if there may be no tests.

johnSchnake avatar May 18 '22 17:05 johnSchnake