unified-memory-framework icon indicating copy to clipboard operation
unified-memory-framework copied to clipboard

Extend FAIL_ON_SKIP to gtest

Open lplewa opened this issue 1 year ago • 0 comments

Problem Statement

The current CMake option UMF_TESTS_FAIL_ON_SKIP is designed to handle skipped tests. However, it does not work with GTEST_SKIP in Google Test. This limits its functionality and does not enforce consistent behavior for skipped tests.

Objective

Extend the UMF_TESTS_FAIL_ON_SKIP option to work with GTEST_SKIP to ensure that skipped tests are treated as failures when this option is enabled.

Solution

Implement a macro to replace GTEST_SKIP with a custom macro (UMF_SKIP). This macro will conditionally either call GTEST_SKIP or fail the test based on the UMF_FAIL_ON_SKIP option.

Implementation Steps

  1. Define the Custom Macro:

Create a new macro UMF_SKIP which will conditionally handle skipped tests based on the UMF_FAIL_ON_SKIP option.

#ifdef UMF_FAIL_ON_SKIP
#define UMF_SKIP GTEST_FAIL() << "UMF_TESTS_FAIL_ON_SKIP Enabled: Treating skip as fail"
#else
#define UMF_SKIP GTEST_SKIP()
#endif
  1. Replace GTEST_SKIP in Tests:

Replace all occurrences of GTEST_SKIP in tests with UMF_SKIP.

  1. Define UMF_FAIL_ON_SKIP Through Compiler Argument:

Ensure the UMF_FAIL_ON_SKIP macro can be defined via a compiler argument (e.g., -DUMF_FAIL_ON_SKIP). Modify the CMake configuration to include this option.

if(UMF_TESTS_FAIL_ON_SKIP)
    add_definitions(-DUMF_FAIL_ON_SKIP)
endif()
  1. Update CI Configuration:

Explicitly exclude tests that are expected to skip on the CI environment in the YAML configuration file for CI pipeline. This is the hardest part, as we do not have option to disable single "gtest" test from YAML level. We did something like this for valgrind tests - but it required separate .sh file for those tests, so better approach is to add extra cmake configuration flags <TEST_NAME>_FILTER and modify add_test function to handle it

lplewa avatar Jul 26 '24 14:07 lplewa