capsule-render
capsule-render copied to clipboard
Sonobuoy results may count incorrectly in some cases
What steps did you take and what happened: Ran e2e tests in parallel with the postprocessor to remove skips. Final result had some xml files with no tests. So the 'k8s test suite' was the leaf node (which is unusual). When counting tests, sonobuoy then reported 9 tests instead of the 5 I expected.
File looks like:
name: e2e-0
status: unknown
meta:
type: summary
items:
- name: sonobuoy_results.yaml
status: ""
meta:
file: results/global/sonobuoy_results.yaml
type: file
items:
- name: junit_01.xml
status: passed
meta:
file: /tmp/sonobuoy/results/junit_01.xml
type: file
items:
- name: Kubernetes e2e suite
status: passed
items:
- name: '[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] patching/updating
a mutating webhook should work [Conformance]'
status: passed
- name: '[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] patching/updating
a validating webhook should work [Conformance]'
status: passed
- name: '[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] listing
mutating webhooks should work [Conformance]'
status: passed
- name: junit_02.xml
status: passed
meta:
file: /tmp/sonobuoy/results/junit_02.xml
type: file
items:
- name: Kubernetes e2e suite
status: passed
items:
- name: '[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] listing
validating webhooks should work [Conformance]'
status: passed
- name: junit_03.xml
status: passed
meta:
file: /tmp/sonobuoy/results/junit_03.xml
type: file
items:
- name: Kubernetes e2e suite
status: passed
items:
- name: '[sig-api-machinery] AdmissionWebhook [Privileged:ClusterAdmin] should
be able to deny attaching pod [Conformance]'
status: passed
- name: junit_04.xml
status: passed
meta:
file: /tmp/sonobuoy/results/junit_04.xml
type: file
items:
- name: Kubernetes e2e suite
status: passed
- name: junit_05.xml
status: passed
meta:
file: /tmp/sonobuoy/results/junit_05.xml
type: file
items:
- name: Kubernetes e2e suite
status: passed
...
Part of the problem is in the post-processor plugin; it removed a bunch of tests and left none of the original leaves. However, the junit processing should think about how to handle the 'suite' in general if there may be no tests.