First class representation of assertion errors in reporting

[laeubi]

Currently one get the following message for a (failed) test_step_finished

test_step_finished {
  test_step_result {
    status: FAILED
    message: "org.opentest4j.AssertionFailedError: expected: <8.0> but was: <3.0>\n\tat org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)\n\tat org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62)\n\tat org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)\n\tat org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)\n\tat org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1124)\n\tat io.cucumber.examples.java.RpnCalculatorSteps.the_result_is(io/cucumber/examples/java/basic_arithmetic.feature:27)\n"
    duration {
      nanos: 2984000
    }
  }
  timestamp {
    seconds: 1619766696
    nanos: 398493000
  }
  test_step_id: "c2692c18-91e6-4a38-8cd8-c40ef71ca67e"
  test_case_started_id: "22500d5c-fb17-464a-8f7e-8afa01342e58"
}

to prevent the consumer from parsing the message is would be good to have the 'expected: <8.0> but was: <3.0>' part split in different fields

Describe the solution you'd like

test_step_finished {
  test_step_result {
    status: FAILED
   expected: "<8.0>",
   actual: "<3.0>",
    message: "org.opentest4j.AssertionFailedError: expected: <8.0> but was: <3.0>\n\tat org.junit.jupiter.api.AssertionUtils.fail(AssertionUtils.java:55)\n\tat org.junit.jupiter.api.AssertionUtils.failNotEqual(AssertionUtils.java:62)\n\tat org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:182)\n\tat org.junit.jupiter.api.AssertEquals.assertEquals(AssertEquals.java:177)\n\tat org.junit.jupiter.api.Assertions.assertEquals(Assertions.java:1124)\n\tat io.cucumber.examples.java.RpnCalculatorSteps.the_result_is(io/cucumber/examples/java/basic_arithmetic.feature:27)\n"
    duration {
      nanos: 2984000
    }
  }
  timestamp {
    seconds: 1619766696
    nanos: 398493000
  }
  test_step_id: "c2692c18-91e6-4a38-8cd8-c40ef71ca67e"
  test_case_started_id: "22500d5c-fb17-464a-8f7e-8afa01342e58"
}

[mpkorstanje]

Moving this to the mono repo. For context:

This is a proposed change to the message protocol. In Java, assertion libraries use the well known org.opentest4j.AssertionFailedError to communicate the expected and actual results in an assertion to various APIs. The proposal would add these well fields to the message protocol when available.

I've got no opinion on this. Just adding additional clarification.

[laeubi]

Thanks for clarification.

The proposal would add these well fields to the message protocol when available.

Correctly, that would be good, to the tools consuming the messages don't need to know if it was java or some other runner...

[aslakhellesoy]

There is an implicit assumption here that an assertion is comparison between two values.

What is the type of value here? Looking at the opentest4j it looks like it can be objects of any type: https://github.com/ota4j-team/opentest4j/blob/master/src/main/java/org/opentest4j/ValueWrapper.java

So if we do this, the actual and expected fields could be 5, "5", {"age": 5}, "2021-04-30T08:18:37.302Z" etc.

I suppose the consumers (reporting tools) could render primitives (strings, numbers, dates) fine. Arrays of primitives could be rendered in a bullet list. But for deeply nested objects we'd have to display them as a tree.

Something to think about...

[luke-hill]

Some of the libraries might work with this, however if we did this, we would need a way of obtaining both the previously unstitched values, and we would have to have reasonable confidence most / all libraries do this (I think they do, but it's just a consideration).

[davidjgoss]

Something to test against if/when we get to this is the assertion library used by Playwright Test, which is often cited by users wanting this, see https://github.com/cucumber/cucumber-js/issues/2418.

Also some overlap with https://github.com/cucumber/react-components/issues/393 since the diff including ANSI colours is often embedded in the Error.message.