WIP chore(ci_visibility): collect module-level line execution and imports

[romainkomorn-exdatadog]

NOTE: still very much a WIP

This explores changes necessary for coverage collection for the Intelligent Test Runner in order to be able to properly capture import-time dependencies on other modules.

To achieve this, the following are introduced:

• optional instrumentation of import statements to capture that a given file imported a given module
• optional capture of execution of module-level lines during coverage collection
• optional reporting of import-time covered lines for modules traversed during regular coverage

The pytest usage of the internal coverage library is updated to use the collection of import-time lines, resulting in expanded coverage for all tests (or suites if using suite-level skipping).

Checklist

• [ ] Change(s) are motivated and described in the PR description
• [ ] Testing strategy is described if automated tests are not included in the PR
• [ ] Risks are described (performance impact, potential for breakage, maintainability)
• [ ] Change is maintainable (easy to change, telemetry, documentation)
• [ ] Library release note guidelines are followed or label changelog/no-changelog is set
• [ ] Documentation is included (in-code, generated user docs, public corp docs)
• [ ] Backport labels are set (if applicable)
• [ ] If this PR changes the public interface, I've notified @DataDog/apm-tees.

Reviewer Checklist

• [ ] Title is accurate
• [ ] All changes are related to the pull request's stated goal
• [ ] Description motivates each change
• [ ] Avoids breaking API changes
• [ ] Testing strategy adequately addresses listed risks
• [ ] Change is maintainable (easy to change, telemetry, documentation)
• [ ] Release note makes sense to a user of the library
• [ ] Author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• [ ] Backport labels are set in a manner that is consistent with the release branch maintenance policy

[datadog-dd-trace-py-rkomorn[bot]]

Datadog Report

Branch report: romain.komorn/CIVIS-9428/add_session_coverage Commit report: 0b535b8 Test service: dd-trace-py

:x: 35 Failed (0 Known Flaky), 175423 Passed, 1139 Skipped, 11h 27m 18.3s Total duration (27m 59.73s time saved)

:x: Failed Tests (35)

This report shows up to 5 failed tests.

• test_cmdi - test_pygoat.py - Details

  Expand for error

  'Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
  

• test_cmdi - test_pygoat.py - Details

  Expand for error

  'Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
  

• test_cmdi - test_pygoat.py

• test_cmdi - test_pygoat.py - Details

  Expand for error

  'Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
  

• test_cmdi - test_pygoat.py - Details

  Expand for error

  'Connection aborted.', ConnectionResetError(104, 'Connection reset by peer'))
  

[pr-commenter[bot]]

Benchmarks

Benchmark execution time: 2024-07-29 14:53:45

Comparing candidate commit 0bb048029f859b50c47ae2ba50c07013e005e5a0 in PR branch romain.komorn/CIVIS-9428/add_session_coverage with baseline commit 08ed1994498a56d81b74083828bd7241c19c9613 in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 214 metrics, 2 unstable metrics.

[datadog-dd-trace-py-rkomorn[bot]]

Datadog Report

Branch report: romain.komorn/CIVIS-9428/add_session_coverage Commit report: 03d74fb Test service: dd-trace-py

:white_check_mark: 0 Failed, 175120 Passed, 1214 Skipped, 10h 2m 11.25s Total duration (13m 56.18s time saved)

[github-actions[bot]]

CODEOWNERS have been resolved as:

tests/coverage/__init__.py                                              @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/import_time_callee.py                      @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/import_time_lib.py                         @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/imported_in_function_lib.py                @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/imports_ns_dot.py                          @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/imports_rp_dot.py                          @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/late_import_const.py                       @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/nested_import_time_lib.py                  @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/normal_import_const.py                     @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/nsa/imports_dotdot_nsa.py                  @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/nsa/imports_dotdot_nsb.py                  @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/nsa/late_import_const.py                   @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/nsa/normal_import_const.py                 @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/nsa/nsa/late_import_const.py               @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/nsa/nsa/normal_import_const.py             @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/nsa/nsa_imports_dot.py                     @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/nsa/nsa_imports_parent.py                  @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/nsa/nsb/late_import_const.py               @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/nsa/nsb/normal_import_const.py             @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/nsb/late_import_const.py                   @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/nsb/normal_import_const.py                 @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/rpa/__init__.py                            @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/rpa/imports_dotdot_nsa.py                  @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/rpa/imports_dotdot_nsb.py                  @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/rpa/late_import_const.py                   @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/rpa/normal_import_const.py                 @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/rpa/rpa/__init__.py                        @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/rpa/rpa/late_import_const.py               @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/rpa/rpa/normal_import_const.py             @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/rpa/rpa_imports_dot.py                     @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/rpa/rpa_imports_parent.py                  @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/rpa/rpb/__init__.py                        @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/rpa/rpb/late_import_const.py               @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/rpa/rpb/normal_import_const.py             @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/rpb/__init__.py                            @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/rpb/late_import_const.py                   @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/included_path/rpb/normal_import_const.py                 @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/test_coverage.py                                         @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/test_coverage_relative_namespace_packages.py             @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/test_coverage_relative_regular_packages.py               @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/utils.py                                                 @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
ddtrace/contrib/pytest/plugin.py                                        @DataDog/ci-app-libraries
ddtrace/internal/ci_visibility/coverage.py                              @DataDog/ci-app-libraries
ddtrace/internal/coverage/code.py                                       @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/coverage/installer.py                                  @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/coverage/instrumentation.py                            @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/coverage/instrumentation_py3_10.py                     @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/coverage/instrumentation_py3_11.py                     @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/coverage/instrumentation_py3_12.py                     @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/coverage/instrumentation_py3_8.py                      @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/coverage/multiprocessing_coverage.py                   @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/coverage/report.py                                     @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/coverage/threading_coverage.py                         @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/module.py                                              @DataDog/debugger-python @DataDog/apm-core-python
pyproject.toml                                                          @DataDog/python-guild
riotfile.py                                                             @DataDog/apm-python
setup.py                                                                @DataDog/python-guild
tests/coverage/test_coverage_multiprocessing.py                         @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
tests/coverage/test_coverage_threading.py                               @DataDog/apm-core-python @DataDog/ci-app-libraries @Datadog/debugger-python
ddtrace/internal/coverage/instrumentation_py3_7.py                      @DataDog/apm-core-python @datadog/ci-app-libraries @Datadog/debugger-python

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 9.89717% with 701 lines in your changes missing coverage. Please review.

Project coverage is 10.49%. Comparing base (c728c68) to head (e8f87f3). Report is 53 commits behind head on main.

Files	Patch %	Lines
...overage/test_coverage_relative_regular_packages.py	0.00%	151 Missing :warning:
...erage/test_coverage_relative_namespace_packages.py	0.00%	139 Missing :warning:
ddtrace/internal/coverage/code.py	13.74%	113 Missing :warning:
...dtrace/internal/coverage/instrumentation_py3_12.py	13.11%	53 Missing :warning:
tests/coverage/test_coverage.py	0.00%	48 Missing :warning:
...dtrace/internal/coverage/instrumentation_py3_11.py	18.18%	36 Missing :warning:
...dtrace/internal/coverage/instrumentation_py3_10.py	18.91%	30 Missing :warning:
ddtrace/internal/coverage/instrumentation_py3_7.py	21.05%	30 Missing :warning:
ddtrace/internal/coverage/instrumentation_py3_8.py	21.05%	30 Missing :warning:
ddtrace/internal/module.py	39.13%	28 Missing :warning:
... and 10 more

Additional details and impacted files

@@             Coverage Diff             @@
##             main    #9609       +/-   ##
===========================================
- Coverage   74.30%   10.49%   -63.82%     
===========================================
  Files        1398     1372       -26     
  Lines      129930   128608     -1322     
===========================================
- Hits        96541    13493    -83048     
- Misses      33389   115115    +81726     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[romainkomorn-exdatadog]

Very interesting idea! Did you by any chance run test-environment with these changes to estimate how much slow down does this approach bring?

TL;DR: we're slightly faster than the original approach, with better functionality.

It's a bit fuzzy because the current strategy using coverage.py does not capture import-time dependencies. I mucked around with some other attempts to capture coverage at import time but pytest's hooks weren't being called at the right time.

That said, our test env results are a tiny bit faster with this approach than with the original coverage.py-based approach. Anecdotally, the gathering of dependencies does come at a cost of about 10% more overhead than without, but overall has ~5-10% less overhead than the current approach.

There's one other aspect that is fuzzy here, and that is that this approach basically makes all tests in a suite have the same dependencies, which makes the behavior much more like suite-level skipping than test-level skipping, and I'll almost certainly end up updating the plugins to do suite-level skipping as a result, and the overhead with suite-level skipping is a little lower than with test-level skipping because we're sending less data over the wire and doing the BFS for dependencies coverage less often.