zio icon indicating copy to clipboard operation
zio copied to clipboard

Generate richer test results

Open swoogles opened this issue 2 years ago • 0 comments

Why

Enable features beyond what SBT provides:

  • Performance Regressions
  • Recognize flaky tests
  • More debugging data when a test fails
  • Serve as input data for a future ZioTest webapp

What

  • Serialize new ZTestResult instances
  • Likely JSON
  • Persist results between runs

First draft of new result type:

case class ZTestResult(
  fullyQualifiedName: String,
  labels: List[String],
  // For grouping results. Cannot reliably be done with labels
  ancestors: List[SuiteId],
  test: Either[TestFailure, TestSuccess],
  annotations: TestAnnotationMap,
  time: Duration,
  // Need to decide how to track this in a way that's not
  // rendered meaningless by different environments
  historicalAverage: Duration,
  suspectedFlaky: Boolean,
  // In addition to real-time streaming output, we could also capture the output
  // from specific tests for later browsing
  output: Chunk[String]
)

Questions:

  • What is our persistence solution?
  • Is there a good way to save CI results from transient pipeline machines?
  • How should we serialize results without bloating dependencies?
    • Should it be JSON?

Currently, zio-test only generates test-reports in the basic XML format expected by SBT. This behavior would remain unchanged:

<?xml version='1.0' encoding='UTF-8'?>
<testsuite hostname="bills-mbp.lan" name="zio.BlockingSpec" tests="8" errors="0" failures="0" skipped="0" time="0.0" timestamp="2022-07-28T22:12:58">
  <properties>
      <property name="java.specification.version" value="17"/>
      ...
  </properties>
  <testcase 
          classname="zio.BlockingSpec" 
          name=" - Make a Blocking Service and verify that - attemptBlocking runs on the blocking thread pool"
          time="0.0"
  >
  </testcase>
  <testcase 
          classname="zio.BlockingSpec" 
          name=" - Make a Blocking Service and verify that - attemptBlocking completes successfully" 
          time="0.0"
  >
  </testcase>
  <system-out><![CDATA[]]></system-out>
  <system-err><![CDATA[]]></system-err>
</testsuite>

swoogles avatar Aug 01 '22 17:08 swoogles