Let TestReporter publish entries with values not being strings

[oprb]

Background: I tried to implement an extension to make it possible measuring the duration of test executions. So I created an extension like the one in the JUnit5 User Guide. Everything seemed fine when I checked the example implementation provided there ("TimingExtension"). But there, the duration is just logged. However, I need to include the results in a report created by implementing a custom TestExecutionListener. Looking at the ExtensionContext interface, I thought I could use the instance provided as an argument to the beforeTestExecution/afterTestExecution methods (defined in BeforeTestExecutionCallback and AfterTestExecutionCallback, respectively) to publish entries containing the time a test is started/is finished. But, at a second glance I saw that the report entries has to be consumed by a EngineExecutionListener, which is, in my case, sadly no supertype of TestExecutionListener. So, there seems no easy way to publish those points in time to my own listener. Sadly, I found no documentation for the EngineExecutionListener in the user guide, so I don't expect the reports are internally received and passed further, such that they are also avaible in execution listeneres. So two questions arise:

• A compound value type holding the start time of test execution and the end time of test execution published through my extension for further use in the listener would be a good fit. Sadly, only publishing the string representation of these times seem to be possible. Therefore, the duration of actual test execution, if not published additionally also as a string representation (which would be kinda redundant in my opinion), has to be computed in the listener after first parsing the time points' string representations. Why isn't it simply possible to publish non-string keys?
• How can I publish results from an extension to an execution listener?

Thanks to you in advance for any response.

[marcphilipp]

Tentatively slated for 5.7 M2 solely for the purpose of team discussion.

[marcphilipp]

We did a little brainstorming during our team call today.

We could potentially add the following to TestExecutionListener:

default void reportDataPublished(TestIdentifier testIdentifier, ReportData<?> data) {
}

interface ReportData<T> {
	LocalDateTime getTimestamp();
	String getName();
	T getPayload();
}

and this to ExtensionContext:

<T> void publishReportData(String name, T payload);

@oprb Would that be sufficient to support your use case?

[marcphilipp]

@baev Would this suit your needs as well?

[mpkorstanje]

This would also be usable for Cucumber. Cumber currently allows users to publish attachments. These don't map well to strings. Using a custom object or map of string-object would solve this.

What is the purpose of the name field? And what are the properties of the name that Consumers should be able to rely upon?

[baev]

Would this suit your needs as well?

Depends on implementation. The current issue we have is that report entries are published to system out in case you run tests via IDEA. So it would be nice to have some internal communication link or storage.

What do you think about building bi-directional communications? It maybe useful to be able to share state managed on platform level with engine extensions.

[marcphilipp]

What do you think about building bi-directional communications? It maybe useful to be able to share state managed on platform level with engine extensions.

I'd say let's cross that bridge when we come to it. For now, we don't have a real use case for that so let's focus on getting this issue resolved first.

[marcphilipp]

Tentatively slated for 5.8 M1 solely for the purpose of team discussion.

[pawel-smuga]

@marcphilipp Is there any plan to resolve this issue? I'm asking because I'm experiencing entries printed to system out a lot.

[beluha]

I also interested in this enhancement. My use cases are:

1. I have extension class implementing LifecycleMethodExecutionExceptionHandler and TestExecutionExceptionHandler interfaces. In all overridden methods I do screenshot of application under test at occurred exception. I would like to pass screenshot as BufferedImage object to the TestExecutionListener (logging execution information into HTML report). For example, end-user can also do screenshots in the test or attach them to the report: testReporter.publishEntry(LogInfo.IMAGE, new Image(imagePath, image));, where Image class is overlay storing path to the file and image itself.
2. Another requirement is to allow to the end-user in the @Test method to call testReporter.publishEntry(Object, Object); because end-user prefer to publish info in more simple format like testReporter.publishEntry(LogInfo.TEST_DESCRIPTION, new TestScriptInfo("Test and verify AutomationProperties", "Alfira Merinova")); where TestScriptInfo represents simple object with test description and script author.
3. It's normal case to log thrown exception in the report. Since I handle exceptions in extension class, I need to call context.publishReportEntry(LogInfo.EXCEPTION, exception); needed to pass exception as it is to the TestExecutionListener reporting class.

Can you please implement solution as ReportEntry(Object, Object) or similar to? Suggested solution in https://github.com/junit-team/junit5/issues/2296#issuecomment-646571351 can also work.

[marcphilipp]

Attaching files such as screenshots or device logs to tests is another use case for this.