Integrate Generators into reporters and CLI

[horenmar]

Unlike Sections, Generators are currently invisible to reporters, and also to the CLI. This means that in test case like this

TEST_CASE() {
    SECTION("a") { ... }
    auto i = GENERATE(1, 2, 3);
    SECTION("b") { ... }
}

the user can ask to run specifically "a" section, or the "b" section 3 times, but they cannot ask to run the test case with i == 2 (whether they want to run a specific section or not).

This then makes errors in generator-based tests harder to debug than they should be (debugging the test case for 17th element out of 23 is hard to set up).

This is an important step towards death tests, which require ability to run specific test in a different process.

[horenmar]

Some design notes for this

• There should be a new listener event for generators.
• To simplify the work to create a custom reporter, we should be sending paired start ... end events. This enables reporters to keep track of active generators without having to do the bookkeeping of tracking each generator's parent section.
  • The name generatorStarting is symmetric with the existing events, and also terrible, because it does not match the actual semantics of GENERATE. generatorEnded is even worse.
  • Current consideration: generatorActivated ... generatorDeactivated.
• The starting event should be sent whenever we visit an expression with GENERATE in it.
  • The ending event can only be sent when the parent section's tracker is closing. We have to ensure that the generator trackers emit the ending events in the proper LIFO order.
• The starting event should pass either the generator interface, or something similar, that allows both getting the index of the current element in the generator, and getting the string representation of that element.
  • The ending event should not send these, as the generator might no longer be valid when the event is fired. Or we have to codify the semantics that the event is always sent before the generator can be deallocated, and make it clear that the generator-interface-pointer should never ever be stored.
  • We should also consider sending in the "index" of the generator that is (de)activated. It is not important, but it might simplify some things... but users will have to maintain their own stack anyway, maybe it is not worth it?
  • Unlike sections, generators do not have user-facing names, so we cannot use those.
• We want to keep backwards compatibility with the existing -c, --section path filtering. To avoid overcomplicating the CLI option, we will support two exclusive path specification modes, so the users can either use -c ... as they do now, or they can use -p, --path in the future, but not both at the same time.
  • The --path specifier will accept compound arguments, such as --path g:2 or --path c:"foo bar" for generator index or section name respectively.
    • Should we use 'c' or 's' (or both) for the section filter? 'c' is consistent with -c argument, -s makes more sense. We can also use both, but that risks user confusion.
    • Should we also allow long names, so that g:2 and generator:2 are equivalent? Also should the long names be fully explicit, e.g. generator-index:2, section-name:"foo bar"?

Example event semantics

This sample test case

TEST_CASE() {
    SECTION("A") { ... }
    auto _ = GENERATE(...);
    SECTION("B") { ... }
}

should emit

testCaseStarting
  testCasePartialStarting -> #0
    sectionStarting -> "test"
      sectionStarting -> "A"
      sectionEnded -> "A"
      generatorActivated
      generatorDeactivated
    sectionEnded -> "test"
  testCasePartialEnded -> #0
  testCasePartialStarting -> #1
    sectionStarting -> "test"
      generatorActivated
        sectionStarting -> "B"
        sectionEnded -> "B"
      generatorDeactivated
    sectionEnded -> "test"
  testCasePartialEnded -> #1
  ... repeated for each generator element ...
testCaseEnded

[mjerabek]

Also, we may want something like

auto foo = NAMED_GENERATE("foo", ...);

for avoiding common patterns

auto foo = GENERATE(...);
CAPTURE(foo);

This will also present the generator name to reporters, so that in cases when we are generating something that we care to be named, we will see a reasonable name there.