CTest test driver

[MathiasMagnus]

As mentioned on WG calls, Stream HPC got the greenlight to upstream the CTest test driver we developed under contract that allows parallel test execution.

Notes on the contribution

• The feature from a user's perspective is thoroughly documented. The CTS doesn't quite do a good job at documenting what each specific test does, but more importantly how and why they do the things they do. We would like to set precedent on improving on that, specifically looking out for new contributors of the CTS.
• Based on WG discussions it became apparent that frequent contributors to the CTS have homegrown scripts that accelerate running the tests. While this initial take on parallel test execution is far from being feature complete, we believe it's a valuable launchpad for future contributions and refinement.

Notes on the implementation

• How are tests hooked up to the CTest driver?
  • Tests are registered statically from CMake script during configuration.
  • To facilitate extracting the list cases, the common cli option listing the tests was mildly tweaked.
  • The initial set of test case registration happened via an uncommitted PowerShell script, but it can be recreated in reasonable amount of time in any scripting language. For completeness sake:

  foreach($TestSuite in (gci .\.vscode\build\ninja-msvc-v143\test_conformance\ -Exclude select | where -prop Mode -Match d).Name) { if(Test-Path .\.vscode\build\ninja-msvc-v143\test_conformance\$TestSuite\Release\test_$TestSuite.exe) {$CMakeFragment = & .\.vscode\build\ninja-msvc-v143\test_conformance\$TestSuite\Release\test_$TestSuite.exe --list | Out-String; $NewFragment = ('set(${MODULE_NAME}_KERNEL_LANG clc)'+"`n"+'set(${MODULE_NAME}_TESTS'+"`n"+$CMakeFragment+")`n`n"+'include(../CMakeCommon.cmake)'); (gc .\test_conformance\$TestSuite\CMakeLists.txt | Out-String).Replace('include(../CMakeCommon.cmake)',$NewFragment) | Set-Content .\test_conformance\$TestSuite\CMakeLists.txt } }
  

• Is this work complete in coverage?
  • No. Most notably the OpenGL, DirectX and extension tests are not added.
• What aspects were taken into account for future use?
  • The resource spec JSON allows adding extra metadata to the devices being tested. I abused this aspect to add additional information, such as what flavor and version of OpenCL is supported by the device and whether the runtime provides an online compiler or an offline compiler is required.
  • A utility script has been provided that adapts clang.exe to be used by the test_harness machinery as an offline compiler. Example for this is also provided in the docs.
  • This work isn't intended to be a replacement for the conformance submission Python script, but they do cover a fair amount of overlapping ground.
  • The static nature of test registration can be enhanced by compiling a clinfo-like utility at configuration time, or users can be given a helper script much like rcoPRIM does. There are many ways of going about making this feature easier or more automatic to use, up to and including decoupling the test definition into a project/CMake invocation of it's own, independent of building the tests.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[MathiasMagnus]

I still have to make CI happy. It's in the "works on my machine and all our cluster" state.