zio
zio copied to clipboard
Generate richer test results
Why
Enable features beyond what SBT provides:
- Performance Regressions
- Recognize flaky tests
- More debugging data when a test fails
- Serve as input data for a future ZioTest webapp
What
- Serialize new
ZTestResult
instances - Likely JSON
- Persist results between runs
First draft of new result type:
case class ZTestResult(
fullyQualifiedName: String,
labels: List[String],
// For grouping results. Cannot reliably be done with labels
ancestors: List[SuiteId],
test: Either[TestFailure, TestSuccess],
annotations: TestAnnotationMap,
time: Duration,
// Need to decide how to track this in a way that's not
// rendered meaningless by different environments
historicalAverage: Duration,
suspectedFlaky: Boolean,
// In addition to real-time streaming output, we could also capture the output
// from specific tests for later browsing
output: Chunk[String]
)
Questions:
- What is our persistence solution?
- Is there a good way to save CI results from transient pipeline machines?
- How should we serialize results without bloating dependencies?
- Should it be JSON?
Currently, zio-test
only generates test-reports in the basic XML format expected by SBT. This behavior would remain unchanged:
<?xml version='1.0' encoding='UTF-8'?>
<testsuite hostname="bills-mbp.lan" name="zio.BlockingSpec" tests="8" errors="0" failures="0" skipped="0" time="0.0" timestamp="2022-07-28T22:12:58">
<properties>
<property name="java.specification.version" value="17"/>
...
</properties>
<testcase
classname="zio.BlockingSpec"
name=" - Make a Blocking Service and verify that - attemptBlocking runs on the blocking thread pool"
time="0.0"
>
</testcase>
<testcase
classname="zio.BlockingSpec"
name=" - Make a Blocking Service and verify that - attemptBlocking completes successfully"
time="0.0"
>
</testcase>
<system-out><![CDATA[]]></system-out>
<system-err><![CDATA[]]></system-err>
</testsuite>