dd-trace-py icon indicating copy to clipboard operation
dd-trace-py copied to clipboard

ci(appsec): mark unreliable test as flaky

Open emmettbutler opened this issue 1 year ago • 3 comments

This change marks as flaky a test that recently failed in main. I don't see a straightforward way to adjust the expectations in that test to match the actual behavior without completely undermining the functional purpose of the test.

Checklist

  • [x] Change(s) are motivated and described in the PR description
  • [x] Testing strategy is described if automated tests are not included in the PR
  • [x] Risks are described (performance impact, potential for breakage, maintainability)
  • [x] Change is maintainable (easy to change, telemetry, documentation)
  • [x] Library release note guidelines are followed or label changelog/no-changelog is set
  • [x] Documentation is included (in-code, generated user docs, public corp docs)
  • [x] Backport labels are set (if applicable)
  • [x] If this PR changes the public interface, I've notified @DataDog/apm-tees.
  • [x] If change touches code that signs or publishes builds or packages, or handles credentials of any kind, I've requested a review from @DataDog/security-design-and-guidance.

Reviewer Checklist

  • [ ] Title is accurate
  • [ ] All changes are related to the pull request's stated goal
  • [ ] Description motivates each change
  • [ ] Avoids breaking API changes
  • [ ] Testing strategy adequately addresses listed risks
  • [ ] Change is maintainable (easy to change, telemetry, documentation)
  • [ ] Release note makes sense to a user of the library
  • [ ] Author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
  • [ ] Backport labels are set in a manner that is consistent with the release branch maintenance policy

emmettbutler avatar Feb 29 '24 21:02 emmettbutler

Datadog Report

Branch report: emmett.butler/appsec-contrib-flake Commit report: 4eaac80 Test service: dd-trace-py

:white_check_mark: 0 Failed, 924 Passed, 2567 Skipped, 24m 17.92s Total duration (52m 51.76s time saved)

Benchmarks

Benchmark execution time: 2024-03-13 14:40:22

Comparing candidate commit 4eaac805c3fa45ae9f46f6afd6d668c31414aa1b in PR branch emmett.butler/appsec-contrib-flake with baseline commit 5a551036638bf41595c7ef874d1099c076c53cfe in branch main.

Found 3 performance improvements and 1 performance regressions! Performance is the same for 196 metrics, 10 unstable metrics.

scenario:flasksimple-appsec-get

  • 🟩 execution_time [-257.203µs; -211.086µs] or [-3.931%; -3.226%]

scenario:flasksimple-appsec-telemetry

  • 🟥 execution_time [+239.469µs; +293.064µs] or [+3.804%; +4.656%]

scenario:sethttpmeta-no-useragentvariant

  • 🟩 max_rss_usage [-726.617KB; -441.972KB] or [-3.289%; -2.001%]

scenario:sethttpmeta-obfuscation-worst-case-explicit-query

  • 🟩 max_rss_usage [-761.152KB; -508.608KB] or [-3.413%; -2.280%]

pr-commenter[bot] avatar Feb 29 '24 22:02 pr-commenter[bot]

@gnufede do you think this is mergeable?

emmettbutler avatar Mar 13 '24 13:03 emmettbutler