crun icon indicating copy to clipboard operation
crun copied to clipboard

test: add new target "coverage"

Open giuseppe opened this issue 1 month ago • 4 comments

add a new target "make coverage" to generate the lcov coverage report for the tests suite.

Most of the changes were done with Claude Code.

It is the first step to get more coverage in our tests.

Summary by Sourcery

Add configurable code coverage support to the build and introduce richer, structured logging and skip reasons in the Python test suite.

Build:

  • Introduce --enable-coverage configure option with gcov/lcov/gcovr integration and corresponding Makefile coverage targets and flags.

Tests:

  • Enhance Python tests to return structured skip reasons and improve diagnostics via a shared logger and additional /tmp mount.
  • Adjust many tests to hide container stderr by default to reduce noise while preserving detailed error logging.
  • Add a tmpfs /tmp mount to the base test configuration and tweak some test setups/cleanup for robustness.

giuseppe avatar Dec 05 '25 20:12 giuseppe

Reviewer's Guide

Adds optional code coverage support (configure flag, compiler/linker flags, and make coverage targets) and refactors the Python test suite to use a shared logger, structured skip reasons, quieter run_and_get_output calls, and more robust TAP-style diagnostics and cleanup behavior.

File-Level Changes

Change Details Files
Introduce configurable code coverage support and make targets driven by gcov/lcov/gcovr.
  • Add --enable-coverage configure flag that wires up GCOV/LCOV/GCOVR detection and exposes COVERAGE_CFLAGS/COVERAGE_LDFLAGS via autotools
  • Apply coverage CFLAGS/LDFLAGS to libcrun, the crun binary, and the testing library when ENABLE_COVERAGE is set
  • Add non-parallel coverage-* make targets (clean/reset/check/html/xml/summary) that run the test suite serially and post-process coverage data, plus hook coverage-clean into clean-local
  • Adjust maint.mk coverage output directory and gen-coverage behavior, and mark coverage-related phony targets as non-parallel
configure.ac
Makefile.am
maint.mk
Standardize Python test logging, error reporting, and TAP output handling across the test suite.
  • Introduce a shared logger in tests_utils configured to emit TAP-style diagnostics to stderr, and export it for use in test modules
  • Replace direct sys.stderr writes and some prints in tests with logger.info/logger.error calls to keep diagnostics consistent and parseable
  • Extend the TAP runner in tests_utils to support timing-based slow-test warnings and richer exception diagnostics (including subprocess return codes, commands, and traces)
tests/tests_utils.py
tests/test_devices.py
tests/test_mounts.py
tests/test_exec.py
tests/test_delete.py
tests/test_resources.py
tests/test_uid_gid.py
tests/test_capabilities.py
tests/test_oci_features.py
tests/test_hostname.py
tests/test_mounts.py
tests/test_oci_features.py
tests/test_cwd.py
tests/test_limits.py
tests/test_seccomp.py
tests/test_bpf_devices.py
tests/test_domainname.py
tests/test_pid.py
tests/test_rlimits.py
tests/test_time.py
tests/test_tty.py
tests/test_hooks.py
tests/test_start.py
tests/test_mempolicy.py
Make tests more automation-friendly by returning structured skip reasons and reducing noisy stderr from helpers.
  • Change many tests to return (77, "reason") instead of bare 77 and adjust the TAP runner to print the reason in the #SKIP annotation
  • Pass hide_stderr=True into run_and_get_output in many tests to prevent command stderr from polluting TAP output while still logging via the shared logger
  • Improve robustness in several tests (e.g., more careful cleanup, safer handling when prerequisites like NUMA hardware, systemd, BPF, or /dev/fuse are missing)
tests/tests_utils.py
tests/test_mempolicy.py
tests/test_start.py
tests/test_devices.py
tests/test_mounts.py
tests/test_exec.py
tests/test_delete.py
tests/test_resources.py
tests/test_uid_gid.py
tests/test_capabilities.py
tests/test_limits.py
tests/test_seccomp.py
tests/test_bpf_devices.py
tests/test_time.py
tests/test_tty.py
tests/test_bpf_devices.py
tests/test_update.py
tests/test_pid_file.py
tests/test_init.c (via tests_init_CFLAGS in Makefile.am)
Tighten and extend container test behavior (tmpfs /tmp, mounts, idmapped mounts, cgroups, etc.) to be more deterministic and debuggable.
  • Add a tmpfs /tmp mount with fixed mode/size to the default base_config to ensure predictable tmp directory behavior inside containers
  • Refine various mount-, idmapped mount-, and device-related tests to use clearer expectations, better cleanup, and more structured logging of mismatches
  • Add a small CFLAGS tweak for tests/init to enable debug info and optimization level suitable for coverage and debugging
tests/tests_utils.py
tests/test_mounts.py
tests/test_devices.py
tests/test_update.py
Makefile.am

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an issue from a review comment by replying to it. You can also reply to a review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull request title to generate a title at any time. You can also comment @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in the pull request body to generate a PR summary at any time exactly where you want it. You can also comment @sourcery-ai summary on the pull request to (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the pull request to resolve all Sourcery comments. Useful if you've already addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull request to dismiss all existing Sourcery reviews. Especially useful if you want to start fresh with a new review - don't forget to comment @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

  • Contact our support team for questions or feedback.
  • Visit our documentation for detailed guides and information.
  • Keep in touch with the Sourcery team by following us on X/Twitter, LinkedIn or GitHub.

sourcery-ai[bot] avatar Dec 05 '25 20:12 sourcery-ai[bot]

Ephemeral COPR build failed. @containers/packit-build please check.

TMT tests failed. @containers/packit-build please check.

fuzzing CI fixed here: https://github.com/containers/crun/pull/1922

giuseppe avatar Dec 09 '25 10:12 giuseppe

@jnovy PTAL

giuseppe avatar Dec 11 '25 15:12 giuseppe

@jnovy thanks! Fixed now

giuseppe avatar Dec 12 '25 13:12 giuseppe

The rest is looking good - it currently covers 29% but it's a good starting point. LGTM

jnovy avatar Dec 12 '25 13:12 jnovy