Catch2 icon indicating copy to clipboard operation
Catch2 copied to clipboard

GENERATE doesn't display which values caused test failures in the test report

Open fschuh opened this issue 3 years ago • 4 comments

Description

The GENERATE macro runs a given test multiple times with different parameters, but the test report doesn't seem to provide context or information on which of those parameters are causing failures in the test.

Additional context

As an example, let's consider the following simple test:

    TEST_CASE("Test with generators")
    {
        int value = GENERATE(1, 3, 5);
        REQUIRE_FALSE(testFunction(value));
    }

If the test fails with generated parameter values 1 and 3, this is what the test report looks like:

-------------------------------------------------------------------------------
Test with generators
-------------------------------------------------------------------------------
C:\TestProject1\Tests1.cpp(131)
...............................................................................

C:\TestProject1\Tests1.cpp(134): FAILED:
  REQUIRE_FALSE( testFunction(value) )
with expansion:
  !true

-------------------------------------------------------------------------------
Test with generators
-------------------------------------------------------------------------------
C:\TestProject1\Tests1.cpp(131)
...............................................................................

C:\TestProject1\Tests1.cpp(134): FAILED:
  REQUIRE_FALSE( testFunction(value) )
with expansion:
  !true

-------------------------------------------------------------------------------

As we can see, there aren't any hints whatsoever that the values 1 and 3 were the reasons of those failures, while 5 was good.
If there isn't a way to have Catch2 display the generated parameters on the report, is this something that would be feasible to add? Are there any reasons why this information doesn't show up in the test report?

fschuh avatar Feb 08 '22 01:02 fschuh

This is a handy feature that I found myself looking for previously too. I don't know what's blocking Catch2 from supporting it currently, but a workaround is to use dynamic sections with manual for-loops:

    TEST_CASE("Test with generators")
    {
        for (int value : { 1, 3, 5 }) {
            DYNAMIC_SECTION("VALUE IS " << value) {
                REQUIRE_FALSE(testFunction(value));
            }
        }
    }

neobrain avatar Feb 09 '22 15:02 neobrain

Another workaround is to use the CAPTURE macro.
However, that adds extra noise to the test code, and you have to remember to use it on every test.
It would be nice if we could get the values to show up in the report without having to add additional boilerplate to the test.

fschuh avatar Feb 09 '22 17:02 fschuh

Are there any updates for this feature, after a year?

tanlin2013 avatar Mar 24 '23 07:03 tanlin2013