Add appsec benchmarks

[estringana]

Description

Reviewer checklist

• [ ] Test coverage seems ok.
• [ ] Appropriate labels assigned.

[codecov-commenter]

Codecov Report

:white_check_mark: All modified and coverable lines are covered by tests. :white_check_mark: Project coverage is 81.07%. Comparing base (31382ec) to head (95aa3f3). :warning: Report is 610 commits behind head on master.

Additional details and impacted files

@@             Coverage Diff              @@
##             master    #2791      +/-   ##
============================================
+ Coverage     81.05%   81.07%   +0.01%     
  Complexity     2517     2517              
============================================
  Files           146      146              
  Lines         14654    14654              
============================================
+ Hits          11878    11880       +2     
+ Misses         2776     2774       -2     

Flag	Coverage Δ
tracer-extension	78.18% <ø> (ø)
tracer-php	82.27% <ø> (+0.01%)	:arrow_up:

Flags with carried forward coverage won't be shown. Click here to find out more. see 1 file with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 31382ec...95aa3f3. Read the comment docs.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• :package: JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[pr-commenter[bot]]

Benchmarks [ profiler ]

Benchmark execution time: 2024-09-02 14:10:17

Comparing candidate commit 9db79ba6457b16f5c0a40bb4cddc24717cdb8f8b in PR branch estringana/add-appsec-benchmarks with baseline commit 05ae6270e82aa0a170b3ce97f7176a2d3d962d06 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 27 metrics, 9 unstable metrics.

[pr-commenter[bot]]

Benchmarks [ tracer ]

Benchmark execution time: 2024-09-25 09:08:18

Comparing candidate commit 95aa3f34b8e7c71ff6395da18158d43d2e7cacb7 in PR branch estringana/add-appsec-benchmarks with baseline commit 31382ec7e2f822cf156883b11e249f4bfd4db72e in branch master.

Found 2 performance improvements and 9 performance regressions! Performance is the same for 167 metrics, 0 unstable metrics.

scenario:EmptyFileBench/benchEmptyFileBaseline-opcache

• 🟥 execution_time [+103.079µs; +235.101µs] or [+3.494%; +7.968%]

scenario:EmptyFileBench/benchEmptyFileOverhead

• 🟩 execution_time [-185.295µs; -67.985µs] or [-5.855%; -2.148%]

scenario:LaravelBench/benchLaravelBaseline

• 🟥 execution_time [+7.138ms; +7.310ms] or [+230.113%; +235.670%]

scenario:LaravelBench/benchLaravelBaseline-opcache

• 🟥 execution_time [+7.287ms; +7.432ms] or [+233.886%; +238.513%]

scenario:LaravelBench/benchLaravelOverhead

• 🟥 execution_time [+7.781ms; +7.992ms] or [+235.894%; +242.277%]

scenario:LaravelBench/benchLaravelOverhead-opcache

• 🟥 execution_time [+7.837ms; +8.052ms] or [+231.866%; +238.216%]

scenario:LogsInjectionBench/benchLogsInfoBaseline-opcache

• 🟩 execution_time [-215.290ns; -166.310ns] or [-12.663%; -9.782%]

scenario:WordPressBench/benchWordPressBaseline

• 🟥 mem_peak [+512.544KB; +512.544KB] or [+7.068%; +7.068%]

scenario:WordPressBench/benchWordPressBaseline-opcache

• 🟥 mem_peak [+198.464KB; +198.464KB] or [+4.415%; +4.415%]

scenario:WordPressBench/benchWordPressOverhead

• 🟥 mem_peak [+2.944MB; +2.944MB] or [+61.087%; +61.087%]

scenario:WordPressBench/benchWordPressOverhead-opcache

• 🟥 mem_peak [+2.688MB; +2.688MB] or [+134.034%; +134.034%]

[pr-commenter[bot]]

Benchmarks [ appsec ]

Benchmark execution time: 2024-09-11 09:53:43

Comparing candidate commit ccd86d54a9f5c75f7a1e79184ebaef6b2cb4a9e9 in PR branch estringana/add-appsec-benchmarks with baseline commit 05ae6270e82aa0a170b3ce97f7176a2d3d962d06 in branch master.

Found 0 performance improvements and 0 performance regressions! Performance is the same for 178 metrics, 0 unstable metrics.

[ddyurchenko]

About - Benchmarks [ tracer ]. Something is completely not right with WordPressBench/benchWordPressOverhead, should not experience such big perf difference. Other 4 scenarios seems to be what usually flaky / not super precise benchmarks look like.

I can increase flakiness threshold for you from 2% to 5%, so the small flaky results are hidden. For most teams we already use 5%, ensuring 2% threshold is possible, but would require quite some engineering time (and in general not sure if it is useful for you -- will you address a regression of size 2% anyhow?).

[ddyurchenko]

I keep wondering what is happening with scenario:WordPressBench/benchWordPressOverhead. It is consistently 80% faster across 5 last runs. It can be that we do something wrong in estringana/add-appsec branch in benchmarking-platform repo during candidate or baseline run.

[pr-commenter[bot]]

Benchmarks

Benchmark execution time: 2024-09-20 08:45:44

Comparing candidate commit 339adfc7dc1eecbe94d48f5cd45dec90ed674976 in PR branch estringana/add-appsec-benchmarks with baseline commit 31382ec7e2f822cf156883b11e249f4bfd4db72e in branch master.

Found 1 performance improvements and 8 performance regressions! Performance is the same for 168 metrics, 1 unstable metrics.

scenario:ContextPropagationBench/benchInject64Bit

• 🟥 execution_time [+158.425ns; +471.575ns] or [+2.014%; +5.994%]

scenario:EmptyFileBench/benchEmptyFileOverhead-opcache

• 🟩 execution_time [-236.273µs; -105.307µs] or [-7.039%; -3.137%]

scenario:LaravelBench/benchLaravelBaseline

• 🟥 execution_time [+7.176ms; +7.392ms] or [+226.330%; +233.156%]

scenario:LaravelBench/benchLaravelOverhead

• 🟥 execution_time [+7.756ms; +7.986ms] or [+230.602%; +237.461%]

scenario:LaravelBench/benchLaravelOverhead-opcache

• 🟥 execution_time [+7.787ms; +8.009ms] or [+226.349%; +232.819%]

scenario:WordPressBench/benchWordPressBaseline

• 🟥 mem_peak [+512.896KB; +512.896KB] or [+7.073%; +7.073%]

scenario:WordPressBench/benchWordPressBaseline-opcache

• 🟥 mem_peak [+198.672KB; +198.672KB] or [+4.419%; +4.419%]

scenario:WordPressBench/benchWordPressOverhead

• 🟥 mem_peak [+2.945MB; +2.945MB] or [+61.093%; +61.093%]

scenario:WordPressBench/benchWordPressOverhead-opcache

• 🟥 mem_peak [+2.689MB; +2.689MB] or [+134.039%; +134.039%]

[estringana]

I keep wondering what is happening with scenario:WordPressBench/benchWordPressOverhead. It is consistently 80% faster across 5 last runs. It can be that we do something wrong in estringana/add-appsec branch in benchmarking-platform repo during candidate or baseline run.

This is fixed. Basically, there was error connecting to the wordpress database. I think this problem was there before but can't understand why it was not spotted