dd-trace-py icon indicating copy to clipboard operation
dd-trace-py copied to clipboard

chore(ci): delay dogfood stage until after benchmarks

Open erikayasuda opened this issue 1 year ago • 2 comments

We originally had dogfood happen before the benchmark stages, which meant that if dogfooding failed (which it often can), we would have stale benchmark results on the PR. This switches the order so we can ensure benchmarks can always run, and will not be blocked by dogfooding.

Checklist

  • [x] Change(s) are motivated and described in the PR description
  • [x] Testing strategy is described if automated tests are not included in the PR
  • [x] Risks are described (performance impact, potential for breakage, maintainability)
  • [x] Change is maintainable (easy to change, telemetry, documentation)
  • [x] Library release note guidelines are followed or label changelog/no-changelog is set
  • [x] Documentation is included (in-code, generated user docs, public corp docs)
  • [x] Backport labels are set (if applicable)
  • [x] If this PR changes the public interface, I've notified @DataDog/apm-tees.

Reviewer Checklist

  • [x] Title is accurate
  • [x] All changes are related to the pull request's stated goal
  • [x] Description motivates each change
  • [x] Avoids breaking API changes
  • [x] Testing strategy adequately addresses listed risks
  • [x] Change is maintainable (easy to change, telemetry, documentation)
  • [x] Release note makes sense to a user of the library
  • [x] Author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
  • [x] Backport labels are set in a manner that is consistent with the release branch maintenance policy

erikayasuda avatar May 24 '24 22:05 erikayasuda

Benchmarks

Benchmark execution time: 2024-06-25 14:44:46

Comparing candidate commit 0b2961897d847bd0af61c11124a9908ca15f3861 in PR branch erikayasuda/delay-dogfood with baseline commit 48df6fa38ae108a49099fc3548a990546f72819f in branch main.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 221 metrics, 9 unstable metrics.

pr-commenter[bot] avatar May 24 '24 22:05 pr-commenter[bot]

Datadog Report

Branch report: erikayasuda/delay-dogfood Commit report: e223577 Test service: dd-trace-py

:x: 1 Failed (0 Known Flaky), 156817 Passed, 19159 Skipped, 6h 18m 5.44s Total Time

:x: Failed Tests (1)

  • test_slow_imports - test_serverless.py - Details

    Expand for error
    xpected status 0, got 1.
    == Captured STDOUT ===
    == End of captured STDOUT ===
    == Captured STDERR ===
    raceback (most recent call last):
     File "/root/project/ddtrace/internal/packages.py", line 250, in is_distribution_available
       import importlib.metadata as importlib_metadata
     File "tests/internal/test_serverless.py", line 125, in find_spec
       raise ImportError(f"module {fullname} was imported!")
    mportError: module importlib.metadata was imported!
    ..