Catch2 icon indicating copy to clipboard operation
Catch2 copied to clipboard

Convert tags to ctest labels

Open conr2d opened this issue 2 years ago • 2 comments

Description

This PR allows converting Catch2 tags to CTest labels.

GitHub Issues

Closes #1590

conr2d avatar Mar 05 '22 12:03 conr2d

Codecov Report

Merging #2383 (a8ccae0) into devel (2ebc041) will not change coverage. The diff coverage is n/a.

@@           Coverage Diff           @@
##            devel    #2383   +/-   ##
=======================================
  Coverage   91.04%   91.04%           
=======================================
  Files         152      152           
  Lines        7313     7313           
=======================================
  Hits         6658     6658           
  Misses        655      655           

codecov[bot] avatar Mar 05 '22 13:03 codecov[bot]

Okay, so in general I am not against, but I have multiple issues with the specific implementation.

  1. Test name does not uniquely identify a TEST_CASE. This means that executing ./tests {test-name} --list-tags can provide tags from multiple different test cases.
  2. Executing the test binary for each separate test is pretty large performance impact.

Both of these can be fixed by taking a harder approach towards registering tests, and extracting both test name and its tags at the same time.

In an ideal world, CMake would have functional xml support, and we could just retrieve the information easily through the xml reporter. We are not in an ideal world, so the next best thing is to run --list-tests, and parse the output. The format can be parsed by using leading indentation as guide, e.g.

{2 spaces}Test Name
{4 spaces}(Optional) Test location
{6 spaces}(Optional) Tags

I am also told that the way script is extended with tags-as-properties will scale badly, but I don't know enough CMake to confirm or deny that.

horenmar avatar Mar 24 '22 15:03 horenmar

The performance of this approach is such that I won't merge it. More notes here: https://github.com/catchorg/Catch2/pull/2696#issuecomment-1593437575

horenmar avatar Jun 15 '23 18:06 horenmar