Allow tests to notify listeners of saved files

[mc1arke]

Tests that save files (screenshots, logs, data extracts etc) currently have no way of notifying external programs where these files have been saved, other than possibly by printing a line to a logger or System.out or by writing to a file that other programs are aware of. It would be nice for tests to be able to notify additional events (e.g file saved) to test listeners so that IDEs, CI servers or any other calling program can link to or load these for a user.

Kohsuke Kawaguchi did write a blog post sometime back about this: http://kohsuke.org/?s=junit+attachment which shows the hack created to allow Jenkins to attach files to test reports.

Actually implementing this is a bit of a challenge since tests don't have visibility of listeners. A couple of options I can think of for approaching this:

1. Allow tests to have a constructor that takes aFileListener parameter for reporting file events to (with FileListener just being a wrapper round a list of TestListeners).
2. Allow test to declare a setFileEventListener(FileListener) method, possibly from a new FileHandlingTest interface
3. Create a FileListenerRule rule that records a list of saved files then throws a FilesSaved exception which runners pick up (similar to how ErrorCollector works).
4. Create a special FileListenerRule rule which runners look for whilst constructing the test and inject any TestListeners into.

Out of these options, 1 and 2 seem to be reverting to JUnit3 style structures so I'm not convinced they're particularly suitable. Option 3 isn't brilliant since it's using exceptions to handle flow control for non error scenarios, wouldn't work with custom runners (failures/errors would be reports for file saves) and may cause issues with some Rules. Option 4 wouldn't report file events for any custom runners that didn't inject listeners into the rule but wouldn't cause failures in this scenario, although does require runners interact directly with a particular rule type which some people may object to.

I'm happy to create some code for the different options but want to know initial opinions about whether anyone wants this feature, and whether anyone has any strong opinions about how this should (or shouldn't) work.

[kcooney]

Why not implement this as a Rule? A rule can know the Description object corresponding to the current test. The Description could be used to match up the file(s) with the test that wrote it. Here's a rough sketch

1. Define a file format that includes for each file an encoding of the Description, the file path, and perhaps the content type
2. Have the IDE or CI server export an environment variable with the path of the file to write
3. Create a Rule that has a method to write the file path and metadata to the file (if the environment variable doesn't exist, perhaps it writes to a temporary directory)
4. Profit

[mc1arke]

@kcooney an interesting idea. I'm not completely convinced about using an environmental variable for this type of action. Consider running a CI server which wants to see files in one place, which executes Ant which wants to see your files in another place: Ant would get precedence on the variable since it set it last so the CI server isn't going to see your files. Obviously this is a bit contrived since Ant would likely contain logic to check for and re-use the current variable contents and hopefully include the notifications in its report which the CI server would parse, but then we get into the realm of whether Maven etc all act the same way.

Getting the rule to be able to report to a listener provided by am external executor seems cleaner (for an external user to use, than for JUnit to develop) and prevents these tools having to try and match up post run logs with in-run listener events.

[kcooney]

That example does seem slightly contrived. Regardless, the CI server or Ant will have to somehow specify the parent directory for these files, to handle things like concurrent runs, cleaning up after runs, etc.

In any case, I was just trying to make suggestions that don't require adding things to the current API. Changing the API will take some time, and then you need to change the IDEs and build tools to understand the new API (but only if the code being built is using a recent-enough version of JUnit). Also, as you pointed out, most possible ways to solve this inside of JUnit wouldn't work for custom runners.

There are a number of possible solutions that can be done without any JUnit changes.

Instead of having the Rule write to a file, it could pass the metadata and test Description to a singleton, and then your listener could read from that singleton. Matching up the logs with the in-run listener event is quite easy, since a test is uniquely identified by a Description object.

One way we solve this at my company is we have a custom listener that sets a thread-local variable before the test is run. That thread-local stores metadata collected during the run. After the listener is notified that the test is done, it stores that data, and ensures it is included in our custom test output format. The advantage to using a thread-local is the metadata could be produced by code that is far removed from the test class.