dd-trace-py
dd-trace-py copied to clipboard
ci(appsec): mark unreliable test as flaky
This change marks as flaky a test that recently failed in main. I don't see a straightforward way to adjust the expectations in that test to match the actual behavior without completely undermining the functional purpose of the test.
Checklist
- [x] Change(s) are motivated and described in the PR description
- [x] Testing strategy is described if automated tests are not included in the PR
- [x] Risks are described (performance impact, potential for breakage, maintainability)
- [x] Change is maintainable (easy to change, telemetry, documentation)
- [x] Library release note guidelines are followed or label
changelog/no-changelogis set - [x] Documentation is included (in-code, generated user docs, public corp docs)
- [x] Backport labels are set (if applicable)
- [x] If this PR changes the public interface, I've notified
@DataDog/apm-tees. - [x] If change touches code that signs or publishes builds or packages, or handles credentials of any kind, I've requested a review from
@DataDog/security-design-and-guidance.
Reviewer Checklist
- [ ] Title is accurate
- [ ] All changes are related to the pull request's stated goal
- [ ] Description motivates each change
- [ ] Avoids breaking API changes
- [ ] Testing strategy adequately addresses listed risks
- [ ] Change is maintainable (easy to change, telemetry, documentation)
- [ ] Release note makes sense to a user of the library
- [ ] Author has acknowledged and discussed the performance implications of this PR as reported in the benchmarks PR comment
- [ ] Backport labels are set in a manner that is consistent with the release branch maintenance policy
Datadog Report
Branch report: emmett.butler/appsec-contrib-flake
Commit report: 4eaac80
Test service: dd-trace-py
:white_check_mark: 0 Failed, 924 Passed, 2567 Skipped, 24m 17.92s Total duration (52m 51.76s time saved)
Benchmarks
Benchmark execution time: 2024-03-13 14:40:22
Comparing candidate commit 4eaac805c3fa45ae9f46f6afd6d668c31414aa1b in PR branch emmett.butler/appsec-contrib-flake with baseline commit 5a551036638bf41595c7ef874d1099c076c53cfe in branch main.
Found 3 performance improvements and 1 performance regressions! Performance is the same for 196 metrics, 10 unstable metrics.
scenario:flasksimple-appsec-get
- 🟩
execution_time[-257.203µs; -211.086µs] or [-3.931%; -3.226%]
scenario:flasksimple-appsec-telemetry
- 🟥
execution_time[+239.469µs; +293.064µs] or [+3.804%; +4.656%]
scenario:sethttpmeta-no-useragentvariant
- 🟩
max_rss_usage[-726.617KB; -441.972KB] or [-3.289%; -2.001%]
scenario:sethttpmeta-obfuscation-worst-case-explicit-query
- 🟩
max_rss_usage[-761.152KB; -508.608KB] or [-3.413%; -2.280%]
@gnufede do you think this is mergeable?