CMake test framework plugin

[LecrisUT]

I want to sketch out some ideas for making tmt-cmake plugin:

• [ ] overall: how to write a tmt plugin? E.g. how can it be included in the .fmf files, how to overwrite it via an editable-like installation, how to minimize coupling with upstream
• [ ] overall: plugin will make sure CMake is available (maybe also ninja for good measure) from package manager
• [x] plan.prepare plugin: runs a configure-build (maybe also install) preset/traditional workflow
  • Source folder is in fmf tree, and build folder is in plan-data
  • Currently will limit that only one CMake prepare step is present
  • All other steps, must check that this step exists and use the build folder from this step (hence why only 1 CMake prepare step currently)
• [ ] test.framework plugin: run ctest for a build environment defined in previous feature
  • Currently considering, this just runs the full ctest suite with the filters defined in plan.discover
• [x] plan.discover plugin: convert test names and ctest labels to tmt tests and tags
  • https://cmake.org/cmake/help/latest/manual/ctest.1.html#show-as-json-object-model
  • Blocked by #2813
  • Currently considering, this step only generates one super-test and the user can use this step to add the ctest filters
• [ ] ~~plan.report plugin: recover junit results from ctest and forward it down~~
  • Need to rethink if this is necessary
• [ ] CMake: make tmt commands and/or python environment available in CMake
• [ ] CMake: add test wrapper to convert tmt-native test to ctest callable tests
• [ ] CMake: expose a tmt subcommand to be executed as part of ctest --build-and-test. E.g. thic could parse the build stdout/stderr, build folder artifacts, etc. as part of a regression test and pass it to something like beakerlib.

[lukaszachy]

Hopefully https://tmt.readthedocs.io/en/stable/code/plugin-introduction.html answers the 'how to write plugin' questions. I don't see the framework documented, so this is https://github.com/teemtee/tmt/blob/main/tmt/frameworks/init.py

Each plugin has 'require' which is added to the other (test) require and installed during 'prepare' step on the guest.

tldr; it isn't in .fmf files but you need to have installed on the runner.

[happz]

Indeed, WRT test framework https://github.com/teemtee/tmt/blob/main/tmt/frameworks/init.py#L38 (https://tmt.readthedocs.io/en/stable/code/autodocs/tmt.frameworks.html#tmt.frameworks.TestFramework) and e.g. https://github.com/teemtee/tmt/blob/main/tmt/frameworks/shell.py

WRT prepare plugin, any would work as an example, e.g. https://github.com/teemtee/tmt/blob/main/tmt/steps/prepare/shell.py

[LecrisUT]

Thanks, what I had in mind is to have a requirement.txt section for the fmf files where the plugins can auto-magically be pulled. Probably always on the plans level. Since they all have to be python based, we can just call pip internally to get the dependencies, with probably a safe-guard that the automation only occurs in venv, and outside of it there is a print-out of where/how to get the dependency from pypi.

One complication in that approach is how to refresh the python environment to load the newly installed packages :thinking:

[lukaszachy]

'auto-magicall' pull would be interesting to have, but I'm afraid that for production CI it will be disabled (or allow-listed) anyway so I'm not sure that it is worth pursuing right now. Lets' start with cmake as a plugin which is pip installed manually (read: on workers as TF or actions or ..)

We have 2 plugins internally and frankly speaking it is pain to keep them working with 3 versions of tmt - the released (pypi, fedora), the main branch (copr teemtee/tmt for early adopters) and PR which modifies (read: breaks) API. Code of those plugins can have lot of hasattr if/else if some release is particular heavy with development.

With that said, having it this repo is easier for the maintainer of such plugin.

[LecrisUT]

I have started the repo: https://github.com/LecrisUT/tmt-cmake and I'll see how it goes. First step is to get the plugin to be loaded and detected when tmt plans show

[LecrisUT]

Seeing the light at the end of the tunnel, the only remaining part to have a minimum plugin is extracting the test results more cleanly and adding more tests. Will soon ping for a bit of review after that.