Feature: basic cmd line interface to execute/exclude single tests and suites

[RalphSteinhagen]

Hi, first thanks for this great unit-testing framework! The more I use this library, the more appreciate its features, and lean and clean coding style! Thanks for sharing and making this public! :+1:

As a feature request, it would be nice if UT could support the following out-of-the-box:

1. execution of a single tests, individual or sub-sets of suites
2. exclusion of single or multiple tests (opposite of the above)
3. export the test results in a machine-readable format that could help/simplify UT integration into IDEs.

all based on and using the command-line interface arguments (i.e. no arguments -> default behaviour: text output).

Since the tests do not strictly require a main() would be nice to explore options like this and/or other portable workarounds to automatically parse the command name and arguments if they are not explicitly passed to the framework.

This might be also useful if multiple tests are distributed over different compilation units where an int main() {} in each compilation unit would collide with each other.

Maybe others could chime in with some additional requests. I'd be willing to provide a prototype/proof-of-concept unless someone else is inclined to do so.

EDIT: If nobody else is opposed, I'd propose reuse the same Catch2 naming/syntax for better recognition and IDE integration, i.e.:

<ut_test_executable_name> [<test name|pattern|tags> ... ] options

with options as in (check-boxes to keep track for subsequent PRs):

• [x] -?, -h, --help display usage information
• [x] -l, --list-tests list all/matching test cases
• [x] -t, --list-tags list all/matching tags
• [x] -s, --success include successful tests in output
• ~-b, --break break into debugger on failure~
• ~-e, --nothrow skip exception tests~
• ~-i, --invisibles show invisibles (tabs, newlines)~
• [ ] -o, --out <filename> output filename
• [x] -r, --reporter <name> reporter to use (defaults to console)
• [ ] -n, --name <name> suite name
• [x] -a, --abort abort at first failure
• [x] -x, --abortx <no. failures> abort after x failures
• [ ] -w, --warn <warning name> enable warnings
• [x] -d, --durations <yes|no> show test durations
• [ ] -D, --min-duration <seconds> show test durations for [...]
• [ ] -f, --input-file <filename> load test names to run from a file
• -#, --filenames-as-tags adds a tag for the filename -> ???
• ~-c, --section <section name> specify section to run~
• ~-v, --verbosity <quiet|normal|high> set output verbosity
• [x] --list-test-names-only list all/matching test cases names only
• [x] --list-reporters list all reporters
• [ ] --order <decl|lex|rand> test case order (defaults to decl)
• [ ] --rng-seed <'time'|number> set a specific seed for random numbers
• [x] --use-colour <yes|no> should output be colourised
• [x] --libidentify report name and version according to libidentify standard
• [ ] --wait-for-keypress <never|start|exit|both> waits for a keypress before exiting
• --benchmark-[...]-> this has been initially proposed here and I'd throw in additionally this as a proposal (2nd PR).

As reporter options Catch2 implements:

• compact: Reports test results on a single line, suitable for IDEs
• console: Reports test results as plain lines of text
• junit: Reports test results in an XML format that looks like Ant's junitreport target
• xml: Reports test results as an XML document

I guess we'd need primarily to implement a default junit or xml test output which should satisfy most IDEs.

N.B. This doesn't mean that the follow-up PR will implement everything of the above but just outlining what would be useful as a whole ... eventually.

[krzysztof-jusiak]

Agree @RalphSteinhagen, that would be very neat and defo a great addition which would simplify the usage. Any help always more than welcome, it would be awesome to have a prototype although I don't see much reasons why the feautre couldn't get in, thanks @RalphSteinhagen!

[jhasse]

IMHO having to manually pass argc and argv to boost::ut::cfg<> would be better than black magic like __attribute__((constructor)).

[RalphSteinhagen]

@krzysztof-jusiak @jhasse updated the proposed command line arguments.

IMHO having to manually pass argc and argv to boost::ut::cfg<> would be better than black magic like attribute((constructor)).

That would certainly be the internal/external API. I'd propose this be used as a fallback option. __attribute__((constructor)) is supported by a wide range of compilers -- notably: gcc, clang, icc -- and to my knowledge only msvc/Windows does not support this.

The issue with this is what to do when developers forget to explicitly pass this to UT and/or when there are multiple unit-test compilation units that are linked to one big test.

Other test-framework solve this by declaring a macro that defines the void main(..) and passing of the command line arguments once using pre-compiler guards.

The auto-catching of the command-line arguments is certainly not a must but would ease/simplify the user-experience.

[jhasse]

Thanks for proposing the cmd line options. For me a simple interface which addresses

1. execution of a single tests, individual or sub-sets of suites
2. exclusion of single or multiple tests (opposite of the above)

would be enough as all the other stuff would make µt not µ anymore, wouldn't it?

and to my knowledge only msvc/Windows does not support this.

IMHO that's already a deal-breaker. And if it means that you need to add

#ifdef _MSC_VER
int main(int argc, char** argv) {
    return boost::ut::cfg_main(&argc, &argv);
}
#endif

somewhere anyway, you might as well drop the ifdef.

The issue with this is what to do when developers forget to explicitly pass this to UT [...]

Developers know about the main function. I don't think they would expect a command line tool to handle options when they let argv be unused. Also documentation could be adjusted, so I don't see this as a problem.

Google Test also works like that btw: https://google.github.io/googletest/reference/testing.html#InitGoogleTest

[...] and/or when there are multiple unit-test compilation units that are linked to one big test.

Then there still can only be one main function, right?

[RalphSteinhagen]

would be enough as all the other stuff would make µt not µ anymore, wouldn't it?

For my part, I'd consider 'µ' as: single-header, only C++20 STL dependencies, and the functionality that is required to integrate UT into an IDE. However, whether this qualifies as µ or not, I'd leave to the lib maintainers.

What I have so far is quite lightweight and only relies on the stdlib/STL. Most of the args are for filtering the suites/tests and execution order which IMO is important. Also, keeping it close to some other established standard is IMO useful for users to transition to UT. I haven't done the junit report target but that would be probably be the larger part of the PR. The rest is rather small and simple.

IMHO that's already a deal-breaker. And if it means that you need to add [..]

A strong statement. There are several slightly different/equivalent functions for msvc that do the same and there are already several statements in UT that differentiate between compilers. Windows historically treated command line options differently and shifted only in recent years closer to Unix/POSIX-like systems. Again, this auto-arg config is optional.

Then there still can only be one main function, right?

Well ... g++ -Wl,--allow-multiple-definition a.cpp b.cpp -o main is your friend.

For large projects, you'd want to keep your unit tests in separate compilation units while developing and having one merged global executable including all. The merging IMO would be out-of-scope for UT because there are several other ways to do this besides the one mentioned above.

[RalphSteinhagen]

@krzysztof-jusiak am a bit more familiar with the UT internals now. Most of the required stuff was/is already there which made the job quite easy. :+1:

I have a basic minimal cmd-line parsing interface (settings stored in struct cfg {..} as static fields), pattern matching, "..."_test filtering, and basic junit reporting in place.

I'd like to keep things as neat, clean, and minimally invasive as possible. I hit a tiny bit of a snag while implementing some of the additional cmd-line features mentioned above where I'd like to get your angle on before continuing:

• test suites do not have names that could be filtered -> may I add something similar to the tests, e.g. "..."_suite?
• while most is constexpr-evaluated and strongly typed (which IMO is good), there are some cases where runtime polymorphisms is required, namely the cmd-line argument-dependent switch of runner<reporter>(). What's your preferred choice of for that? std::variant<>? virtual inheritance? ...? It's mainly needed for the part [[maybe_unused]] inline auto cfg = ....
• there are no global suite/test lists: tests are always executed in the order they are defined. This has some impact w.r.t. --order and easier filtering and/or repeating (e.g. flaky) tests. If this shall be supported -- as a proposal -- would it be OK to introduce a global list, and then filter/execute the suites/tests?

More as a templating style comment: many of the templates are defined using SFINAE which works but yields pages of compiler errors in case the user made a mistake. Using C++20 concept's as constraints might help, would document, and provide users nicer early feedback on what they need to implement if they want to customise the test, runner, reporter, printer, etc. What do you think?

_{P.S. I like simple/enforced formatting rules and this is not a code-correctness thing: the clang-format enforced 80-character line length limit increases the cognitive complexity of the internal UT code and makes, for example, some of the template definitions hard to read as the line-breaks do not follow a logic C++ language semantics. An additional indication for this is that UT seems to need to use quite a bit of short-hand notations for types and arguments -- what feels like -- to circumvent the enforced 80-character line limit. Again, not a code-correctness thing but maybe something for future consideration.}

[krzysztof-jusiak]

Thank you @RalphSteinhagen, awesome stuff!

• "...:_suite would be nice, totally for it, not sure how easy it is though as suites are usually in global scope where ""_test is not allowed :thinking:
• static_polymorphism, variant , either is fine (just not virtual dispatch)
• yep, totally fine :+1:

Yes, concepts can be introduced now, when first implemention happend they weren't supported yet but defo we should switch.

Thanks again, fantastic job :clap:

[R2RT]

Hey, I've actually tried to use junit reporter with MSVC, but I found no way to manually call parse in main before the reporter is initialized and tests are collected, since all of it happens in static construction.

For now I ended up with using __argc and __argv macros, but I agree with @jhasse that having access to boost::ut::cfg_main(&argc, &argv); call would be nice.

I don't know how to guarantee that my inject_args's constructor is invoked before the reporter's constructor.

#ifdef _MSC_VER
struct inject_args {
    inject_args() {
        ::boost::ut::detail::cfg::largc = __argc;
        ::boost::ut::detail::cfg::largv = const_cast<const char**>(__argv);
    }
};
static inject_args _inject_args;
#endif

[RalphSteinhagen]

@R2RT, you should already have access to those functions.

The core problem/issue is that most of the suite definitions are already initialised and executed before 'main()' is called due to the static definitions. I am thinking about how one could get cleanly/neatly around that but I haven't found a solution yet ...

Any idea/PR is welcome...

[R2RT]

Yeah, I know I can call them, but no idea how to pass args before static initialization.

So far I found there is run_report.cpp, which, I guess, should be starting point to achieve it. At least from user perspective.