Nested Tests

[ghost]

Nested tests

Expected Behavior

A file test.cpp contains

#include "ut.hpp"

int main() {
  using namespace boost::ut;

  test("set") = []{
    test("one") = []{ expect(1==1); };
    test("two") = []{ expect(2==2); };
  };
}

Compilation and execution is

$ g++ -std=c++20 -Wall -Wextra -pedantic   -c -o test.o test.cpp
$ g++ -std=c++20 -Wall -Wextra -pedantic -o test test.o
$ ./test
All tests passed (2 asserts in 2 tests)

Actual Behavior

All other things being equal,

$ ./test
All tests passed (2 asserts in 1 tests)

Steps to Reproduce the Problem

Project Directory Structure

./ut-test/
  |
  ├── ut.hpp
  |
  └── test.cpp

See Expected Behavior.

Specifications

OS: bento/ubuntu-20.04
Compiler: gcc version 10.0.1 (Ubuntu 10-20200411-0ubuntu1)
          g++ -> x86_64-linux-gnu-gcc-10

Further Implications

In BDD syntax, the standard output is undesirable. For example, something like

#include "ut.hpp"

int main() {
  using namespace boost::ut;

  feature("sort") = []{
    scenario("small arrays") = [] {
      given("trivial arrays") = [] {
        when("the array is empty") = [] { ... };
        when("the array is of size 1") = [] { ... };
      };
      given("array of size 2") = { ... };
      given("array of size 3") = { ... };
      ⋮  
    };
    scenario("significant arrays") = [] {
      given("array of size 20") = { ... }
      ⋮  
    };
    ⋮  
    scenario("mega arrays") = [] {
      ...
    };
  };
}

can result in the output

$ ./test
All tests passed (23192 asserts in 1 tests)

Suggestion

Add an optional parameter to the test function that will cause it to be counted as a test:

... auto test = [](const auto name, const bool isTest) { ...

Then, one could apply it as follows

#include "ut.hpp"

int main() {
  using namespace boost::ut;

  test("set") = []{
    test("one", true) = []{ expect(1==1); };
    test("two", true) = []{ expect(2==2); };
  };

  feature("sort") = []{
    scenario("small arrays") = [] {
      given("trivial arrays") = [] {
        when("the array is empty", true) = [] { ... };
        when("the array is of size 1", true) = [] { ... };
        ⋮  
      };
      ⋮  
    };
    ⋮  
  };
}

Employ a more elegant solution or update the tutorial if my understanding is off.

Thanks!

[JohelEGP]

How about counting all of them?

#include "ut.hpp"

int main() {
  using namespace boost::ut;

  test("set") = []{
    expect(sizeof(1) == 4); // Sanity check before nested tests.
    test("one") = []{ expect(1==1); };
    test("two") = []{ expect(2==2); };
  };
}

$ ./test
All tests passed (3 asserts in 3 tests)

[ghost]

That would be a nice option.

[ytimenkov]

What I would like to see is not that only "one' or "two" are executed, but either "one" or "two" are executed on a given path.

Imagine that both tests share some setup logic which needs to be replicated:

test("vector") = [] {
    // prepare common part
    std::vector v{1,2,3};
    test("erase") = [&] {
        v.erase(v.begin());
        expect(v.size() == 2_u);
    };
    test("insert") = [&] {
        v.insert(v.begin(), 0);
       expect(v.size() == 4_u);
    };
};

In this case there should be only 2 tests: vector.erase and vector.insert, however currently UT treats them as one test:

Running "vector"...
 "erase"...
 "insert"...
  <source>:16:FAILED [3 == 4]
FAILED

===============================================================================
tests:   1 | 1 failed
asserts: 2 | 1 passed | 1 failed

It might seem redundant for a simple vector, but tests may share quite complex initialization logic which diverts quite deep in a flow. In other words copying variables into lambda is not always in option.

Currently one should either copy-paste initialization code (which is really hard to maintain when system evolves) or move common steps into functions (which makes it much harder to see what the test really does).

My guess is that technically this can be done by marking tests as leaves (maybe with new keyword) and do only one leaf in a time (per level or maybe per some label). I.e. run to first leaf, mark it as done, then skip/count other leaves. If there are any, restart the suite, skipping already "done" leaves.

It should be semantically equivalent to

    test("vector") = [](std::string_view which) {
        // prepare common part
        std::vector v{1, 2, 3};
        if (which == "erase")
        {
            test("erase") = [&] {
                v.erase(v.begin());
                expect(v.size() == 2_u);
            };
        }
        if (which == "insert")
        {
            test("insert") = [&] {
                v.insert(v.begin(), 0);
                expect(v.size() == 4_u);
            };
        }
    } | std::vector{"erase"sv, "insert"sv};

All tests passed (2 asserts in 2 tests)