spec icon indicating copy to clipboard operation
spec copied to clipboard

Standardizing `outcome` type and values in doc, schema, and examples

Open xbcsmith opened this issue 1 year ago • 1 comments

In core.md we state that outcome is a String enumerated as success, error or failure.

Examples where we differ are as follows:

In testing-events.md the outcome of the testSuite execution, one of pass, fail, cancel, error

conformance/testcaserun_finished.json: "outcome": "pass", conformance/testsuiterun_finished.json: "outcome": "fail", conformance/pipelinerun_finished.json: "outcome": "failure", conformance/taskrun_finished.json: "outcome": "failure",

schemas/testcaserunfinished.json: "outcome": {"type": "string", "enum": [ "pass", "fail", "cancel", "error"]},

The outcome of this issue should be standardization on success, error or failure as a type String for all outcome.

xbcsmith avatar Jul 08 '24 15:07 xbcsmith

Thanks @xbcsmith - I agree we should standardise where possible however, different subjects may have different semantics: in case of test cases, failure refers to an assertion that is not verified, while error refers to an unexpected error, typically something that lies within the infrastructure required for the execution of the test.

afrittoli avatar Jul 22 '24 14:07 afrittoli