WIP chore(ci_visibility): add Early Flake Detection to pytest v2 plugin

[romainkomorn-exdatadog]

This adds Early Flake Detection logic for the v2 pytest plugin.

Worthwhile notes:

• Tests that fail during the initial setup or teardown are not retried
• Counts displayed by pytest come from the terminalreporter's stats attribute, so the pytest_terminal_summary hook makes some changes to move items around
• Errors displayed by pytest also come from the terminalreport's stats attribute, so error reports are temporarily copied into failed tests and then removed
• Exit status cannot be modified by pytest_terminal_summary, so it is instead modified during pytest_session_finish.
• CIVisibility is now disabled in pytest_unconfigure() because some information (eg: EFD enablement status) is required during `pytest_terminal_summary)

No release note is added since this is still pre-release.

Checklist

• [ ] PR author has checked that all the criteria below are met
• The PR description includes an overview of the change
• The PR description articulates the motivation for the change
• The change includes tests OR the PR description describes a testing strategy
• The PR description notes risks associated with the change, if any
• Newly-added code is easy to change
• The change follows the library release note guidelines
• The change includes or references documentation updates if necessary
• Backport labels are set (if applicable)

Reviewer Checklist

• [ ] Reviewer has checked that all the criteria below are met
• Title is accurate
• All changes are related to the pull request's stated goal
• Avoids breaking API changes
• Testing strategy adequately addresses listed risks
• Newly-added code is easy to change
• Release note makes sense to a user of the library
• If necessary, author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
• Backport labels are set in a manner that is consistent with the release branch maintenance policy

[pr-commenter[bot]]

Benchmarks

Benchmark execution time: 2024-10-28 10:02:13

Comparing candidate commit 2533e7cb3e2a77843aeb62574e97aa1843f90a5d in PR branch romain.komorn/SDTEST-869/efd_retries_in_pytest with baseline commit 83ce1188f5f0987457664bc461283f06570760dc in branch main.

Found 1 performance improvements and 0 performance regressions! Performance is the same for 387 metrics, 2 unstable metrics.

scenario:iast_aspects-rsplit_aspect

• 🟩 execution_time [-237.685ns; -214.680ns] or [-12.526%; -11.313%]

[github-actions[bot]]

CODEOWNERS have been resolved as:

ddtrace/contrib/pytest/_efd_utils.py                                    @DataDog/ci-app-libraries
ddtrace/contrib/pytest/_retry_utils.py                                  @DataDog/ci-app-libraries
tests/contrib/pytest/test_pytest_efd.py                                 @DataDog/ci-app-libraries
ddtrace/contrib/pytest/_plugin_v2.py                                    @DataDog/ci-app-libraries
ddtrace/contrib/pytest/_utils.py                                        @DataDog/ci-app-libraries
ddtrace/contrib/pytest/constants.py                                     @DataDog/ci-app-libraries
ddtrace/contrib/pytest/plugin.py                                        @DataDog/ci-app-libraries
ddtrace/internal/ci_visibility/api/_base.py                             @DataDog/ci-app-libraries
ddtrace/internal/ci_visibility/api/_session.py                          @DataDog/ci-app-libraries
ddtrace/internal/ci_visibility/api/_test.py                             @DataDog/ci-app-libraries
ddtrace/internal/ci_visibility/recorder.py                              @DataDog/ci-app-libraries
ddtrace/internal/test_visibility/_efd_mixins.py                         @DataDog/ci-app-libraries
hatch.toml                                                              @DataDog/python-guild
tests/ci_visibility/api/fake_runner_efd_all_pass.py                     @DataDog/ci-app-libraries
tests/ci_visibility/api/fake_runner_efd_faulty_session.py               @DataDog/ci-app-libraries
tests/ci_visibility/api/fake_runner_efd_mix_fail.py                     @DataDog/ci-app-libraries
tests/ci_visibility/api/fake_runner_efd_mix_pass.py                     @DataDog/ci-app-libraries
tests/ci_visibility/test_efd.py                                         @DataDog/ci-app-libraries
tests/ci_visibility/util.py                                             @DataDog/ci-app-libraries
tests/contrib/pytest/test_coverage_per_suite.py                         @DataDog/ci-app-libraries
tests/contrib/pytest/test_pytest.py                                     @DataDog/ci-app-libraries

[datadog-dd-trace-py-rkomorn[bot]]

Datadog Report

Branch report: romain.komorn/SDTEST-869/efd_retries_in_pytest Commit report: 2533e7c Test service: dd-trace-py

:white_check_mark: 0 Failed, 1286 Passed, 0 Skipped, 31m 27.72s Total duration (6m 42.13s time saved)