[Question] How to run a test both at runtime and compile time

[jan-moeller]

With constexpr code becoming more and more prevalent, I find it increasingly annoying that one has to essentially duplicate all tests if you want to test both the runtime and the compiletime path. In ut, is there a way to run the same test twice, once in a constexpr context, and once at regular runtime?

[80Ltrumpet]

I'm curious about this. If a compile-time test passes, what additional benefit is there to performing the same test at run-time? What do you currently do to test both paths? An example would be helpful.

[jan-moeller]

Often, constexpr code is very inefficient at runtime, which is why an implementation might choose to (partially) delegate to different implementation strategies depending on std::is_constant_evaluated() / if consteval. In this case, both paths need to be tested, but the API is the same. A good reason to run the same test against both implementations. Even if you know that your implementation doesn't do that, I would argue it's better to test both code paths: Because if you don't, in two weeks someone is going to add some code that depends on std::is_constant_evaluated(), and one of the paths will have a bug and you won't know about it.

Typically, the way to write tests that test both paths is to write your entire test code in a lambda, and then run that lambda twice. This, however, is often complicated in practice, since test frameworks are not really meant to be executed at compiletime. Plus, you end up writing a lot of very similar lambdas, often duplicating many lines of test code.

This is mostly experience from other unit test frameworks, since I am not super familiar with ut yet. But, for example, let's say I'm testing std::vector (constexpr in c++20) with catch. Then I would write tests like this:

TEST_CASE("std::vector") {
    constexpr auto test = []{
        std::vector<int>v;
        v.push_back(0);
        return v;
    };
    REQUIRE(test().size() == 1);
    STATIC_REQUIRE(test().size() == 1);
}

ut seems to be in a good position to make this much less painful, since everything is based around lambdas already. However, I was unable to find a way to get it to execute the lambda twice.

[krzysztof-jusiak]

There is constant for that which may help?

• https://github.com/boost-ext/ut/blob/master/example/tmp.cpp

    "std::vector"_test = [] {
         constexpr auto test = []{
            std::vector<int>v;
            v.push_back(0);
            return v;
        };

        expect(1_u == std::size(test()));
        expect(constant<1_u == std::size(test())>);
    };

Full example

• https://godbolt.org/z/48xEsYn7c

[jan-moeller]

@krzysztof-jusiak Right, that's the same kind of workaround I was doing with Catch above. But, what would be really cool is if the following just worked:

"std::vector"_test_both_at_runtime_and_compiletime = [] {
    std::vector<int>v;
    v.push_back(0);
    expect(1_u == std::size(v));
};

Duplicating all assertions is obviously less than ideal, and the only way to make it better is using macros. And I guess making macros unnecessary is one of the key selling points of ut... so yeah, it would be cool to have a feature like that.