vscode-cmake-tools
vscode-cmake-tools copied to clipboard
microsoft
Native coverage integration for the CTest test controller
This change addresses item #4040
This changes visible behavior and documentation
The following changes are proposed:
- Integrate native vscode coverage analysis for the CTest test controller using the VSCode Testing API
Other Notes/Information
This implementation only adds support for GCOV/LCOV based coverage data, but the implementation could be expanded in the future to support additional coverage data.
There are no unit/integration tests added, as it's unclear if the CMake Tools maintainers would be interested in this implementation. If the interest is there, I would be willing to add tests. I have tested the implementation however, and it does work for me.
Some screenshots of this working on your end locally, would also be great to put in the description of the PR.
This looks pretty interesting, so I believe we'd be willing to consider taking this addition into the code, but we would definitely want tests.
Glad to hear! And I agree that tests make sense to include.
From my reading, it looks like this is the test coverage tool for GCC, is that correct? Or does this work for other compilers?
This implementation supports lcov coverage data files. lcov is a graphical front-end for GCC's coverage testing tool gcov. It's worth noting however that other compiler toolchains such as llvm clang are able to produce lcov coverage data files, for example from clang's documentation:
This document explains how to use clang’s source-based code coverage feature. It’s called “source-based” because it operates on AST and preprocessor information directly. This allows it to generate very precise coverage data.
Clang ships two other code coverage implementations: - SanitizerCoverage - A low-overhead tool meant for use alongside the various sanitizers. It can provide up to edge-level coverage. - gcov - A GCC-compatible coverage implementation which operates on DebugInfo. This is enabled by -ftest-coverage or --coverage.
However, if this were to be merged, from my point of view one could also welcome adding support for other coverage data types in addition to the lcov format.
What exactly does this do? I see that you've added another ctest run helper, as well as settings for what target to build before and after .What does running with this coverage do? Why do we need to specify targets to be built before and after?
The settings for targets to be build before and after the tests are executed with coverage are intended for the user to specify utility targets in their CMake setup that prepares for a run with coverage as well as generating the appropriate coverage data files. See "Recommended procedure when capturing data for a test case:" in https://man.archlinux.org/man/lcov.1.en. Essentially, the user is expected to have utility targets that does something similar to this procedure. This is not something I would think the extension can handle, since it will probably look different for different projects. Therefore, it makes sense for the project to have utility targets that would have handle these operations. The extension essentially only handles step 2 (from the recommended procedure) and then the parsing of the files from the cmake.coverageInfoFiles setting (that is also added in this PR) to add coverage information to the editor.
Some screenshots of this working on your end locally, would also be great to put in the description of the PR.
Sure, I'll provide some screenshots or GIFs to show you how it works in the near future as I also work on adding tests.
Let me know if there any more things I should clarify, or if there are any considerations regarding the design of the implementation.
@TSonono Great! Thanks for the explanation. I'll wait on reviewing this further until tests are added, and then it could be useful to add some of the context you just mentioned, into a comment somewhere in the code you added. It doesn't have to be overly verbose, just enough so that when people come back to this for any implementation around this can easily understand. Thanks!
I've now added tests and some rationale notes to the code. Finally, I've added a GIF to the description showing how it works. @gcampbell-msft
@TSonono I'm hitting issues trying to run tests locally now, I'd like to confirm something, is this supported on Windows?
And then therefore, we should only have the CMakeLists.txt modifications for lcov apply to non-windows?
@TSonono I'm hitting issues trying to run tests locally now
I opened a PR #4351 that should hopefully resolve your issue, please try it.
I'd like to confirm something, is this supported on Windows?
The coverage implementation itself is as long as you have access to a compiler which can output gcov based coverage. Regarding the changes to the single-root-UI test project, I was requiring lcov to be found, which is incorrect since this project is also used in Windows. There are windows versions of lcov, but I wouldn't expect most people having them, and therefore it doesn't make sense to require in this test project. One other approach I could have taken would have been to utilize the gcovr pip module instead of lcov, which would have removed the need to use lcov in this test, but I think the change I did in #4351 should be sufficient.