karate
karate copied to clipboard
Mismatched Step Titles and Missing Tags in Reports After Retrying Scenarios
Summary
Two issues have been observed after retrying failed scenarios in a test suite with a Scenario Outline that includes multiple Examples tables:
- The tags from the Examples tables are lost in the final report.
- The titles in the report are mismatched. See the screenshot in the attachment section for an illustrative example.
Steps to Reproduce
- Clone the Repository with the example which will produce the bug https://github.com/Owlwasrowk/karate-template
- Run the test runner using Maven with the command
mvn clean test
. There is used a random flakyness to illustrate the behaviour. - Observe the generated report for missing tags and mismatched titles.
Expected Behavior
After retrying failed scenarios and updating the report:
- The tags from the Examples tables should remain associated with the corresponding scenarios.
- The titles in the report should correctly match the scenarios that were retried.
Actual Behavior
After retrying failed scenarios and updating the results:
- The tags from the Examples tables are missing in the final report.
- There is a mismatch in the report titles, where a failed scenario from one Examples table incorrectly shows as another scenario from a different table.
Environment
- Karate Version: 1.4.1
- Java Version: openjdk 11.0.14.1 2022-02-08 LTS
Additional Information
- The mismatch in titles and missing tags only occur after the retry logic is executed.
- The initial report generated before the retry does not have these issues.
Relevance of the Bug
The tags in our Karate test scenarios are not merely for organizational purposes; they serve a critical role in integrating our test automation framework with our test management system. Each tag corresponds to a unique identifier in Xray, which allows us to map automated test results directly to the relevant test cases in our management system. The loss of tags from the Examples tables after retrying scenarios disrupts this integration.
Solution Hints
The problem with the titles appears to be related to how the updateResults
method in the Suite
class compares existing scenarios to determine if they are equal. The comparison logic, which relies only on the section index and the example index, can erroneously consider different scenarios as equal under certain conditions.
Mismatched Titles
The updateResults
method (here) uses the equals
method of the Scenario
class to match scenarios. The isEqualTo
method (here) checks if the section index and the example index are the same. This can lead to a situation where different scenarios from separate Example tables are incorrectly identified as the same because they share the same index positions within their respective tables.
A potential solution would be to refine the comparison logic to include a more distinctive identifier that takes into account the unique context of each scenario, ensuring that scenarios from different Example tables are not conflated.
Missing Tags
The issue with the missing tags after retrying scenarios is still under investigation.
Attachments
-
karate-reports.zip: The
karate-reports
folder containing the generated report after the retry, showcasing the reported bug. -
Screenshot from the Report where we see, that the Example Table Tags are missing, Example 3 is missing and Example 6 is duplicated.
@Owlwasrowk thanks for the details. Karate doesn't yet support retries and this issue is a great example of what all edge cases come in the way. the best option here is for someone to go through the details (I think you have done some good analysis already) and contribute a PR
@ptrthomas Thank you for your prompt response. I understand that Karate does not currently support retries out-of-the-box, and the complexities that come with implementing such a feature.
I appreciate your suggestion to contribute a PR to address this issue. The missing tags may be related to the Suite
object not retaining information about the scenario's originating example table during retries.
If you have any starting points or insights that could guide the development of a solution, I would be grateful for your advice.
@Owlwasrowk there is a developer guide that may help: https://github.com/karatelabs/karate/wiki/Developer-Guide
but anyone with Java experience should be able to dive in. I think you are spot on with the equals
implementation of scenario. we are missing the concept of what I propose to call "examplesParentIndex", ideally I would rename "examplesIndex" to "examplesRowIndex" and use "examplesIndex" for what's missing
here is where example index is inited: https://github.com/karatelabs/karate/blob/v1.4.1/karate-core/src/main/java/com/intuit/karate/core/ScenarioOutline.java#L109
and here is where examples are created when parsing a feature: https://github.com/karatelabs/karate/blob/v1.4.1/karate-core/src/main/java/com/intuit/karate/core/FeatureParser.java#L282
and now if you add one more component to the isEquals
check, this should work