junit-team
Allow `ConsoleLauncher` to redirect STDOUT and STDERR to files instead of the console
My test suites are large, and write a lot of stuff to output as they go along (mostly from the code under test rather than the tests themselves). I run them using ConsoleLauncher. The tree report at the end of the run is useful. But the torrent of console output is usually not: it is very helpful to have output captured when diagnosing failures, but seeing reams out output for passing tests is not.
I believe that JUnit can capture standard output and error. It would be very useful if there was a flag i could give to ConsoleLauncher to tell it to do this, and write the results to one or more files, instead of the console. The simplest thing would be to write all output to a single file (whose contents would be the same as what i currently see in the console). Significantly more useful would to write the output of each test to its own file, to make it easier to find. Perhaps tests which write no output could not create files at all, not sure.
A potential quirk here is that much of this output is written using a logging framework (usually java.util.logging) with an appender which writes to standard output. It would be possible to configure logging differently in tests, but i would prefer not to have to do that - just capturing output as it is written with the default logging configuration is greatly preferable to me.
One thing that wouldn't be useful to me is separate files for standard output and standard error. There is not usually a huge semantic difference between the two, whereas the relative order of messages written across both is usually very important, and separating the files would discard that.
Is there already some way to do this? Would it be possible for me to write some sort of extension to do this, which i could use without modifying my test sources? If not, would this make a good feature?
@tomwhoiscontrary Do you have a concrete proposal how this would be configured? Via new options on the ConsoleLauncher? Is there another CLI tool we could take inspiration from wrt. naming?
Well, my first thought is that if there is some way i could do this myself (a plug-in sort of thing), then i could play around with it and come back with a more concrete suggestion.
The simplest thing that would address my use case would be a flag to ConsoleLauncher like:
--redirect-streams=FILE Specify a path to a file to which standard output and error from tests will be redirected
I'm not very confident about the name. Sadly i can't think of any other tools which support this. Perhaps the methods in java.lang.ProcessBuilder are a precedent?
I think we should have separate options for stdout and stderr. Taking ProcessBuilder as an example, --redirect-output and --redirect-error would make sense. WDYT?
That would be fine for me - as long as i can redirect both to the same file!
Sure, I think we can make that work by checking if both are set to the same file.
I am still interested in working on this! Just don't have a lot of time for it.
I started by doing a bit of reading and thinking.
Bits of the JDK where we should either capture output, or document that we can't capture output, etc:
- System.out + System.err
- java.io.Console
- java.util.logging.ConsoleHandler
- System.setOut + System.setErr
- Threads used inside tests
Gradle captures output from tests, so it might be a good benchmark to try to do whatever Gradle does. I tried to find out how it does whatever it does, but the Gradle codebase is absolutely labyrinthine; i think the key point is DefaultStandardOutputRedirector, but i'm not sure.
Also interesting is the RedirectStdOutAndErr JUnit rule they use in testing.
~~I might get a chance to actually try some code next weekend ...~~ Just remembered there's a film festival on!
Output capturing is already implemented in JUnit (as report entries): https://junit.org/junit5/docs/current/user-guide/#running-tests-capturing-output
I think we should try reusing that in some form but avoid the indirection of capturing it as strings.
I am interested in working on this problem. I am new to open-source contribution but has experience of working in java. @marcphilipp and @tomwhoiscontrary, Can I look into this issue?
@ShwetaliShimangaud I have not worked on this at all, so please go ahead.
I've tried using output capturing as a ReportEntry, I'm getting the report in the method reportingEntryPublished in a TestExecutionListener, this listener is loaded from the ServiceLoader, the problem is I don't how can I pass data like the directory where I should put the file(s) in, or if we take the route of redirecting all of stdout to a single file like this --redirect-output=FILE I don't know how can I pass the name of the file from the command line to the execution listener, also I don't know how can I avoid capturing outputs as a String, any help would be highly appreciate it.
Update: we can deactivate a TestExecutionListener that's loaded from a ServiceLoader using a configuration parameter : https://junit.org/junit5/docs/current/user-guide/#launcher-api-listeners-custom-deactivation
This can help us deactivate our TestExecutionListener that redirects output if no redirection CLI argument is found.
