Consolidated report for retry runs

[grasshopper7]

Is your feature request related to a problem? Please describe. When there is a retry runner the report generated by the plugin contains duplicate lines for the same scenarios which had failed initially and may have passed\failed on rerun. This is a pain as the report does not reflect the true picture. A pretty frequent situation with Selenium's element not found exceptions.

Describe the solution you'd like Scenarios which are available multiple times should be filtered and sorted such that only the latest one is taken into consideration in the report.

A class to filter the latest Element objects from the Report list can be called before the below line of code CluecumberReportPlugin.

https://github.com/trivago/cluecumber-report-plugin/blob/b13b2edd2d3a647a3c5830f3946b072c40c6e659/plugin-code/src/main/java/com/trivago/cluecumber/CluecumberReportPlugin.java#L116

The Report objects would be collected in a Map with the key as a combination of the feature file uri and the line number of the scenario. The value will be a list of the appropriate Report & Element combinations. Remove entries for which the value list size is less than 2.

Map<ReportURIElementLineContainer, List<ReportElementContainer>> uriRepElemMap = new HashMap<>();
private static class ReportURIElementLineContainer {
	final String uri;
	final int line;
}
private static class ReportElementContainer {
	final Report report;
	final Element element;
}

Each values in the map can be sorted on the basis of the starttime timestamp available on the Element object. For each the first and latest Element and also the original Report can be determined. Remove the first Element from the original Report and add the latest Element to it. Something like below.

List<Report> removeReport = new ArrayList<>();
uriRepElemMap.forEach((k, v) -> {
	Collections.sort(v, Comparator.comparing(re -> re.element.getStartDateTime(), Comparator.reverseOrder()));
	Element removalElement = v.get(v.size() - 1).element;
	Element additionElement = v.get(0).element;
	Report originalReport = v.remove(v.size() - 1).report;

	originalReport.getElements().remove(removalElement);
	originalReport.getElements().add(additionElement);

	removeReport.addAll(v.stream().map(re -> re.report).collect(Collectors.toList()));
});
reports.removeAll(removeReport); 

This can be controlled by using a retry flag with default false value, which can be setup in the pom plugin section.

[bischoffdev]

I have to disagree here because we frequently run the same scenario multiple times and we need this information in the report. Also, Cluecumber shows the exact information that is included in the json file. Omitting something because of the same uri/name is actually falsifying this information. It is a fact that this scenario failed once and passed on the second run so I would not want to lose this data.

[grasshopper7]

Fair enough, point understood. Can this be implemented with a toggle on\off option in the report. Select it and only the latest scenario execution results are displayed. Else the default display of all execution results.

[bischoffdev]

Yes, a configuration option is a good idea actually. I'll keep this in mind.

[himanshujane]

Hello @bischoffdev - I see that this feature will be really useful when we dont care about first run and like to override report by second run results. Any plans to add this feature soon?

This feature is available in this reporting plugin and called:: reducingMethod: String, one of: MERGE_FEATURES_BY_ID, MERGE_FEATURES_WITH_RETEST, SKIP_EMPTY_JSON_FILES, HIDE_EMPTY_HOOKS

[bischoffdev]

I will look at this again

[gdonati78]

FYI @grasshopper7 the new feature added in the latest version v.3.5.0 might suit your needs