dd-trace-java icon indicating copy to clipboard operation
dd-trace-java copied to clipboard

Published 20 hours ago • verified publisher icon DataDog

Add debugger benchmarks

Open jpbempel opened this issue 1 year ago • 2 comments

What Does This Do

benchmarks are run only when a file is modified on paths under dd-java-agent/agent-debugger/*

Motivation

Benchmarks run for every debugger PR to assess improvement/regression/neutral

Additional Notes

Jira ticket: DEBUG-2440

jpbempel avatar Jun 12 '24 15:06 jpbempel

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master jpbempel/benchmarks
git_commit_date 1719942483 1719944214
git_commit_sha 179ce68ab6 e650df895a
release_version 1.37.0-SNAPSHOT~179ce68ab6 1.35.0-SNAPSHOT~e650df895a
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1719947283 1719947283
ci_job_id 560225352 560225352
ci_pipeline_id 38179100 38179100
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 1 performance regressions! Performance is the same for 48 metrics, 14 unstable metrics.

scenario Δ mean execution_time candidate mean execution_time baseline mean execution_time
scenario:startup:petclinic:tracing:Remote Config worse
[+22.754µs; +71.568µs] or [+3.374%; +10.612%]		 721.577µs 674.416µs
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.35.0-SNAPSHOT~e650df895a, baseline=1.37.0-SNAPSHOT~179ce68ab6

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.07 s) : 0, 1069760
Total [baseline] (10.434 s) : 0, 10433825
Agent [candidate] (1.072 s) : 0, 1072119
Total [candidate] (10.442 s) : 0, 10441859
section appsec
Agent [baseline] (1.186 s) : 0, 1186201
Total [baseline] (10.535 s) : 0, 10534943
Agent [candidate] (1.182 s) : 0, 1181584
Total [candidate] (10.61 s) : 0, 10609828
section iast
Agent [baseline] (1.178 s) : 0, 1178261
Total [baseline] (10.832 s) : 0, 10831571
Agent [candidate] (1.169 s) : 0, 1169480
Total [candidate] (10.739 s) : 0, 10738864
section profiling
Agent [baseline] (1.271 s) : 0, 1271075
Total [baseline] (10.621 s) : 0, 10621315
Agent [candidate] (1.269 s) : 0, 1268998
Total [candidate] (10.652 s) : 0, 10652409
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.07 s -
Agent appsec 1.186 s 116.441 ms (10.9%)
Agent iast 1.178 s 108.501 ms (10.1%)
Agent profiling 1.271 s 201.315 ms (18.8%)
Total tracing 10.434 s -
Total appsec 10.535 s 101.118 ms (1.0%)
Total iast 10.832 s 397.746 ms (3.8%)
Total profiling 10.621 s 187.49 ms (1.8%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.072 s -
Agent appsec 1.182 s 109.465 ms (10.2%)
Agent iast 1.169 s 97.361 ms (9.1%)
Agent profiling 1.269 s 196.879 ms (18.4%)
Total tracing 10.442 s -
Total appsec 10.61 s 167.97 ms (1.6%)
Total iast 10.739 s 297.006 ms (2.8%)
Total profiling 10.652 s 210.551 ms (2.0%) 
gantt
    title petclinic - break down per module: candidate=1.35.0-SNAPSHOT~e650df895a, baseline=1.37.0-SNAPSHOT~179ce68ab6

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (669.969 ms) : 0, 669969
BytebuddyAgent [candidate] (662.948 ms) : 0, 662948
GlobalTracer [baseline] (306.623 ms) : 0, 306623
GlobalTracer [candidate] (315.379 ms) : 0, 315379
AppSec [baseline] (50.089 ms) : 0, 50089
AppSec [candidate] (50.514 ms) : 0, 50514
Remote Config [baseline] (674.416 µs) : 0, 674
Remote Config [candidate] (721.577 µs) : 0, 722
Telemetry [baseline] (7.59 ms) : 0, 7590
Telemetry [candidate] (7.639 ms) : 0, 7639
section appsec
BytebuddyAgent [baseline] (679.192 ms) : 0, 679192
BytebuddyAgent [candidate] (676.631 ms) : 0, 676631
GlobalTracer [baseline] (299.218 ms) : 0, 299218
GlobalTracer [candidate] (297.65 ms) : 0, 297650
AppSec [baseline] (153.924 ms) : 0, 153924
AppSec [candidate] (153.706 ms) : 0, 153706
IAST [baseline] (21.539 ms) : 0, 21539
IAST [candidate] (22.548 ms) : 0, 22548
Remote Config [baseline] (634.763 µs) : 0, 635
Remote Config [candidate] (967.575 µs) : 0, 968
Telemetry [baseline] (7.747 ms) : 0, 7747
Telemetry [candidate] (6.901 ms) : 0, 6901
section iast
BytebuddyAgent [baseline] (786.139 ms) : 0, 786139
BytebuddyAgent [candidate] (780.08 ms) : 0, 780080
GlobalTracer [baseline] (296.542 ms) : 0, 296542
GlobalTracer [candidate] (293.661 ms) : 0, 293661
AppSec [baseline] (47.684 ms) : 0, 47684
AppSec [candidate] (46.821 ms) : 0, 46821
IAST [baseline] (26.68 ms) : 0, 26680
IAST [candidate] (26.587 ms) : 0, 26587
Remote Config [baseline] (577.688 µs) : 0, 578
Remote Config [candidate] (592.202 µs) : 0, 592
Telemetry [baseline] (7.081 ms) : 0, 7081
Telemetry [candidate] (8.485 ms) : 0, 8485
section profiling
BytebuddyAgent [baseline] (667.031 ms) : 0, 667031
BytebuddyAgent [candidate] (668.279 ms) : 0, 668279
GlobalTracer [baseline] (390.728 ms) : 0, 390728
GlobalTracer [candidate] (387.682 ms) : 0, 387682
AppSec [baseline] (51.844 ms) : 0, 51844
AppSec [candidate] (51.113 ms) : 0, 51113
Remote Config [baseline] (667.373 µs) : 0, 667
Remote Config [candidate] (779.031 µs) : 0, 779
Telemetry [baseline] (7.364 ms) : 0, 7364
Telemetry [candidate] (7.374 ms) : 0, 7374
ProfilingAgent [baseline] (96.054 ms) : 0, 96054
ProfilingAgent [candidate] (96.752 ms) : 0, 96752
Profiling [baseline] (96.078 ms) : 0, 96078
Profiling [candidate] (96.776 ms) : 0, 96776
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.35.0-SNAPSHOT~e650df895a, baseline=1.37.0-SNAPSHOT~179ce68ab6

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.076 s) : 0, 1075638
Total [baseline] (8.593 s) : 0, 8593222
Agent [candidate] (1.064 s) : 0, 1063743
Total [candidate] (8.58 s) : 0, 8580271
section iast
Agent [baseline] (1.177 s) : 0, 1177113
Total [baseline] (9.006 s) : 0, 9006158
Agent [candidate] (1.17 s) : 0, 1170070
Total [candidate] (9.018 s) : 0, 9017844
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.186 s) : 0, 1186356
Total [baseline] (9.04 s) : 0, 9039589
Agent [candidate] (1.176 s) : 0, 1176343
Total [candidate] (9.016 s) : 0, 9015919
section iast_TELEMETRY_OFF
Agent [baseline] (1.18 s) : 0, 1179755
Total [baseline] (9.019 s) : 0, 9018660
Agent [candidate] (1.167 s) : 0, 1167344
Total [candidate] (9.019 s) : 0, 9019491
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.076 s -
Agent iast 1.177 s 101.475 ms (9.4%)
Agent iast_HARDCODED_SECRET_DISABLED 1.186 s 110.718 ms (10.3%)
Agent iast_TELEMETRY_OFF 1.18 s 104.117 ms (9.7%)
Total tracing 8.593 s -
Total iast 9.006 s 412.936 ms (4.8%)
Total iast_HARDCODED_SECRET_DISABLED 9.04 s 446.367 ms (5.2%)
Total iast_TELEMETRY_OFF 9.019 s 425.439 ms (5.0%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.064 s -
Agent iast 1.17 s 106.328 ms (10.0%)
Agent iast_HARDCODED_SECRET_DISABLED 1.176 s 112.601 ms (10.6%)
Agent iast_TELEMETRY_OFF 1.167 s 103.601 ms (9.7%)
Total tracing 8.58 s -
Total iast 9.018 s 437.573 ms (5.1%)
Total iast_HARDCODED_SECRET_DISABLED 9.016 s 435.648 ms (5.1%)
Total iast_TELEMETRY_OFF 9.019 s 439.22 ms (5.1%) 
gantt
    title insecure-bank - break down per module: candidate=1.35.0-SNAPSHOT~e650df895a, baseline=1.37.0-SNAPSHOT~179ce68ab6

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (674.025 ms) : 0, 674025
BytebuddyAgent [candidate] (655.505 ms) : 0, 655505
GlobalTracer [baseline] (307.671 ms) : 0, 307671
GlobalTracer [candidate] (314.909 ms) : 0, 314909
AppSec [baseline] (50.621 ms) : 0, 50621
AppSec [candidate] (50.479 ms) : 0, 50479
Remote Config [baseline] (750.41 µs) : 0, 750
Remote Config [candidate] (711.662 µs) : 0, 712
Telemetry [baseline] (7.688 ms) : 0, 7688
Telemetry [candidate] (7.67 ms) : 0, 7670
section iast
BytebuddyAgent [baseline] (784.154 ms) : 0, 784154
BytebuddyAgent [candidate] (780.357 ms) : 0, 780357
GlobalTracer [baseline] (296.443 ms) : 0, 296443
GlobalTracer [candidate] (293.867 ms) : 0, 293867
AppSec [baseline] (47.359 ms) : 0, 47359
AppSec [candidate] (46.486 ms) : 0, 46486
IAST [baseline] (28.129 ms) : 0, 28129
IAST [candidate] (27.87 ms) : 0, 27870
Remote Config [baseline] (561.942 µs) : 0, 562
Remote Config [candidate] (583.079 µs) : 0, 583
Telemetry [baseline] (6.972 ms) : 0, 6972
Telemetry [candidate] (7.613 ms) : 0, 7613
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (790.494 ms) : 0, 790494
BytebuddyAgent [candidate] (784.793 ms) : 0, 784793
GlobalTracer [baseline] (298.495 ms) : 0, 298495
GlobalTracer [candidate] (295.552 ms) : 0, 295552
AppSec [baseline] (47.854 ms) : 0, 47854
AppSec [candidate] (46.885 ms) : 0, 46885
IAST [baseline] (27.423 ms) : 0, 27423
IAST [candidate] (26.702 ms) : 0, 26702
Remote Config [baseline] (600.908 µs) : 0, 601
Remote Config [candidate] (586.646 µs) : 0, 587
Telemetry [baseline] (7.837 ms) : 0, 7837
Telemetry [candidate] (8.366 ms) : 0, 8366
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (785.99 ms) : 0, 785990
BytebuddyAgent [candidate] (779.735 ms) : 0, 779735
GlobalTracer [baseline] (297.331 ms) : 0, 297331
GlobalTracer [candidate] (294.308 ms) : 0, 294308
AppSec [baseline] (47.537 ms) : 0, 47537
AppSec [candidate] (51.326 ms) : 0, 51326
IAST [baseline] (27.64 ms) : 0, 27640
IAST [candidate] (21.16 ms) : 0, 21160
Remote Config [baseline] (631.033 µs) : 0, 631
Remote Config [candidate] (582.117 µs) : 0, 582
Telemetry [baseline] (6.984 ms) : 0, 6984
Telemetry [candidate] (6.87 ms) : 0, 6870

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-07-02T18:38:40 2024-07-02T18:45:28
git_branch master jpbempel/benchmarks
git_commit_date 1719942483 1719944214
git_commit_sha 179ce68ab6 e650df895a
release_version 1.37.0-SNAPSHOT~179ce68ab6 1.35.0-SNAPSHOT~e650df895a
start_time 2024-07-02T18:38:27 2024-07-02T18:45:15
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1719946271 1719946271
ci_job_id 560225353 560225353
ci_pipeline_id 38179100 38179100
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 16 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.35.0-SNAPSHOT~e650df895a, baseline=1.37.0-SNAPSHOT~179ce68ab6
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.343 ms) : 1325, 1362
.   : milestone, 1343,
appsec (1.708 ms) : 1684, 1731
.   : milestone, 1708,
appsec_no_iast (1.726 ms) : 1702, 1750
.   : milestone, 1726,
iast (1.47 ms) : 1447, 1493
.   : milestone, 1470,
profiling (1.487 ms) : 1463, 1512
.   : milestone, 1487,
tracing (1.475 ms) : 1450, 1499
.   : milestone, 1475,
section candidate
no_agent (1.348 ms) : 1329, 1367
.   : milestone, 1348,
appsec (1.735 ms) : 1711, 1759
.   : milestone, 1735,
appsec_no_iast (1.692 ms) : 1667, 1717
.   : milestone, 1692,
iast (1.486 ms) : 1462, 1510
.   : milestone, 1486,
profiling (1.488 ms) : 1463, 1512
.   : milestone, 1488,
tracing (1.459 ms) : 1435, 1484
.   : milestone, 1459,
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.343 ms [1.325 ms, 1.362 ms] -
appsec 1.708 ms [1.684 ms, 1.731 ms] 364.292 µs (27.1%)
appsec_no_iast 1.726 ms [1.702 ms, 1.75 ms] 382.687 µs (28.5%)
iast 1.47 ms [1.447 ms, 1.493 ms] 126.432 µs (9.4%)
profiling 1.487 ms [1.463 ms, 1.512 ms] 143.647 µs (10.7%)
tracing 1.475 ms [1.45 ms, 1.499 ms] 131.235 µs (9.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.348 ms [1.329 ms, 1.367 ms] -
appsec 1.735 ms [1.711 ms, 1.759 ms] 387.053 µs (28.7%)
appsec_no_iast 1.692 ms [1.667 ms, 1.717 ms] 343.593 µs (25.5%)
iast 1.486 ms [1.462 ms, 1.51 ms] 137.817 µs (10.2%)
profiling 1.488 ms [1.463 ms, 1.512 ms] 139.594 µs (10.4%)
tracing 1.459 ms [1.435 ms, 1.484 ms] 110.914 µs (8.2%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.35.0-SNAPSHOT~e650df895a, baseline=1.37.0-SNAPSHOT~179ce68ab6
    dateFormat X
    axisFormat %s
section baseline
no_agent (367.384 µs) : 348, 387
.   : milestone, 367,
iast (474.317 µs) : 453, 496
.   : milestone, 474,
iast_FULL (548.218 µs) : 527, 569
.   : milestone, 548,
iast_GLOBAL (509.186 µs) : 487, 531
.   : milestone, 509,
iast_HARDCODED_SECRET_DISABLED (476.485 µs) : 456, 497
.   : milestone, 476,
iast_INACTIVE (452.448 µs) : 430, 474
.   : milestone, 452,
iast_TELEMETRY_OFF (466.463 µs) : 446, 487
.   : milestone, 466,
tracing (447.387 µs) : 426, 468
.   : milestone, 447,
section candidate
no_agent (368.761 µs) : 348, 389
.   : milestone, 369,
iast (476.246 µs) : 455, 497
.   : milestone, 476,
iast_FULL (545.913 µs) : 525, 567
.   : milestone, 546,
iast_GLOBAL (502.543 µs) : 481, 524
.   : milestone, 503,
iast_HARDCODED_SECRET_DISABLED (478.54 µs) : 458, 499
.   : milestone, 479,
iast_INACTIVE (454.297 µs) : 432, 476
.   : milestone, 454,
iast_TELEMETRY_OFF (471.834 µs) : 450, 493
.   : milestone, 472,
tracing (439.729 µs) : 419, 460
.   : milestone, 440,
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 367.384 µs [348.122 µs, 386.647 µs] -
iast 474.317 µs [453.055 µs, 495.579 µs] 106.933 µs (29.1%)
iast_FULL 548.218 µs [526.982 µs, 569.454 µs] 180.834 µs (49.2%)
iast_GLOBAL 509.186 µs [487.175 µs, 531.198 µs] 141.802 µs (38.6%)
iast_HARDCODED_SECRET_DISABLED 476.485 µs [455.526 µs, 497.444 µs] 109.101 µs (29.7%)
iast_INACTIVE 452.448 µs [430.469 µs, 474.427 µs] 85.064 µs (23.2%)
iast_TELEMETRY_OFF 466.463 µs [445.794 µs, 487.132 µs] 99.078 µs (27.0%)
tracing 447.387 µs [426.469 µs, 468.305 µs] 80.003 µs (21.8%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 368.761 µs [348.178 µs, 389.344 µs] -
iast 476.246 µs [455.32 µs, 497.171 µs] 107.485 µs (29.1%)
iast_FULL 545.913 µs [524.616 µs, 567.21 µs] 177.152 µs (48.0%)
iast_GLOBAL 502.543 µs [481.098 µs, 523.989 µs] 133.782 µs (36.3%)
iast_HARDCODED_SECRET_DISABLED 478.54 µs [457.597 µs, 499.483 µs] 109.779 µs (29.8%)
iast_INACTIVE 454.297 µs [432.464 µs, 476.131 µs] 85.536 µs (23.2%)
iast_TELEMETRY_OFF 471.834 µs [450.441 µs, 493.226 µs] 103.072 µs (28.0%)
tracing 439.729 µs [419.207 µs, 460.25 µs] 70.967 µs (19.2%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master jpbempel/benchmarks
git_commit_date 1719942483 1719944214
git_commit_sha 179ce68ab6 e650df895a
release_version 1.37.0-SNAPSHOT~179ce68ab6 1.35.0-SNAPSHOT~e650df895a
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1719946664 1719946664
ci_job_id 560225354 560225354
ci_pipeline_id 38179100 38179100
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.35.0-SNAPSHOT~e650df895a, baseline=1.37.0-SNAPSHOT~179ce68ab6
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.464 ms) : 1453, 1476
.   : milestone, 1464,
appsec (2.25 ms) : 2213, 2286
.   : milestone, 2250,
iast (1.987 ms) : 1943, 2030
.   : milestone, 1987,
iast_GLOBAL (2.037 ms) : 1992, 2081
.   : milestone, 2037,
profiling (1.879 ms) : 1844, 1914
.   : milestone, 1879,
tracing (1.855 ms) : 1821, 1890
.   : milestone, 1855,
section candidate
no_agent (1.471 ms) : 1459, 1483
.   : milestone, 1471,
appsec (2.209 ms) : 2175, 2243
.   : milestone, 2209,
iast (1.965 ms) : 1923, 2006
.   : milestone, 1965,
iast_GLOBAL (2.005 ms) : 1963, 2046
.   : milestone, 2005,
profiling (1.844 ms) : 1811, 1876
.   : milestone, 1844,
tracing (1.841 ms) : 1809, 1874
.   : milestone, 1841,
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.464 ms [1.453 ms, 1.476 ms] -
appsec 2.25 ms [2.213 ms, 2.286 ms] 785.736 µs (53.7%)
iast 1.987 ms [1.943 ms, 2.03 ms] 522.336 µs (35.7%)
iast_GLOBAL 2.037 ms [1.992 ms, 2.081 ms] 572.417 µs (39.1%)
profiling 1.879 ms [1.844 ms, 1.914 ms] 415.22 µs (28.4%)
tracing 1.855 ms [1.821 ms, 1.89 ms] 391.201 µs (26.7%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.471 ms [1.459 ms, 1.483 ms] -
appsec 2.209 ms [2.175 ms, 2.243 ms] 737.861 µs (50.2%)
iast 1.965 ms [1.923 ms, 2.006 ms] 493.527 µs (33.5%)
iast_GLOBAL 2.005 ms [1.963 ms, 2.046 ms] 533.489 µs (36.3%)
profiling 1.844 ms [1.811 ms, 1.876 ms] 372.506 µs (25.3%)
tracing 1.841 ms [1.809 ms, 1.874 ms] 370.132 µs (25.2%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.35.0-SNAPSHOT~e650df895a, baseline=1.37.0-SNAPSHOT~179ce68ab6
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.717 s) : 14717000, 14717000
.   : milestone, 14717000,
appsec (15.106 s) : 15106000, 15106000
.   : milestone, 15106000,
iast (18.841 s) : 18841000, 18841000
.   : milestone, 18841000,
iast_GLOBAL (17.857 s) : 17857000, 17857000
.   : milestone, 17857000,
profiling (15.365 s) : 15365000, 15365000
.   : milestone, 15365000,
tracing (14.871 s) : 14871000, 14871000
.   : milestone, 14871000,
section candidate
no_agent (15.37 s) : 15370000, 15370000
.   : milestone, 15370000,
appsec (15.056 s) : 15056000, 15056000
.   : milestone, 15056000,
iast (19.008 s) : 19008000, 19008000
.   : milestone, 19008000,
iast_GLOBAL (17.817 s) : 17817000, 17817000
.   : milestone, 17817000,
profiling (14.978 s) : 14978000, 14978000
.   : milestone, 14978000,
tracing (15.106 s) : 15106000, 15106000
.   : milestone, 15106000,
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.717 s [14.717 s, 14.717 s] -
appsec 15.106 s [15.106 s, 15.106 s] 389.0 ms (2.6%)
iast 18.841 s [18.841 s, 18.841 s] 4.124 s (28.0%)
iast_GLOBAL 17.857 s [17.857 s, 17.857 s] 3.14 s (21.3%)
profiling 15.365 s [15.365 s, 15.365 s] 648.0 ms (4.4%)
tracing 14.871 s [14.871 s, 14.871 s] 154.0 ms (1.0%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.37 s [15.37 s, 15.37 s] -
appsec 15.056 s [15.056 s, 15.056 s] -314.0 ms (-2.0%)
iast 19.008 s [19.008 s, 19.008 s] 3.638 s (23.7%)
iast_GLOBAL 17.817 s [17.817 s, 17.817 s] 2.447 s (15.9%)
profiling 14.978 s [14.978 s, 14.978 s] -392.0 ms (-2.6%)
tracing 15.106 s [15.106 s, 15.106 s] -264.0 ms (-1.7%)

pr-commenter[bot] avatar Jun 12 '24 15:06 pr-commenter[bot]

Debugger benchmarks

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
ci_job_date 1719946019 1719946437
end_time 2024-07-02T18:48:14 2024-07-02T18:55:11
git_branch master jpbempel/benchmarks
git_commit_sha 179ce68ab6 e650df895a69b27db9e6b0e7c3fe51e561dc7776
start_time 2024-07-02T18:47:00 2024-07-02T18:53:58
See matching parameters
Baseline Candidate
ci_job_id 560225358 560225358
ci_pipeline_id 38179100 38179100
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
git_commit_date 1719944214 1719944214

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 8 metrics, 7 unstable metrics.

See unchanged results
scenario Δ mean agg_http_req_duration_min Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p75 Δ mean agg_http_req_duration_p99 Δ mean throughput
scenario:noprobe unstable
[-45.555µs; +33.979µs] or [-16.915%; +12.617%]		 unstable
[-65.572µs; +41.477µs] or [-21.212%; +13.417%]		 unstable
[-79.631µs; +50.118µs] or [-24.508%; +15.425%]		 unstable
[-840.459µs; +272.644µs] or [-92.034%; +29.856%]		 unstable
[-97.762op/s; +169.861op/s] or [-3.666%; +6.370%]
scenario:basic same same same unstable
[-42499.753ns; +43018.180ns] or [-6.922%; +7.006%]		 unstable
[-155.438op/s; +155.438op/s] or [-6.218%; +6.218%]
scenario:loop unsure
[-53.540µs; -30.781µs] or [-0.439%; -0.253%]		 unsure
[-103.275µs; -32.754µs] or [-0.833%; -0.264%]		 unsure
[-177.760µs; -12.106µs] or [-1.415%; -0.096%]		 same same
Request duration reports for reports 
gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (309.134 µs) : 247, 372
.   : milestone, 309,
basic (312.871 µs) : 299, 326
.   : milestone, 313,
loop (12.405 ms) : 12374, 12436
.   : milestone, 12405,
section candidate
noprobe (297.086 µs) : 261, 333
.   : milestone, 297,
basic (314.752 µs) : 305, 325
.   : milestone, 315,
loop (12.337 ms) : 12303, 12372
.   : milestone, 12337,
  • baseline results
Scenario Request median duration [CI 0.99]
noprobe 309.134 µs [246.535 µs, 371.732 µs]
basic 312.871 µs [299.368 µs, 326.373 µs]
loop 12.405 ms [12.374 ms, 12.436 ms]
  • candidate results
Scenario Request median duration [CI 0.99]
noprobe 297.086 µs [261.092 µs, 333.08 µs]
basic 314.752 µs [304.574 µs, 324.93 µs]
loop 12.337 ms [12.303 ms, 12.372 ms]

pr-commenter[bot] avatar Jun 13 '24 10:06 pr-commenter[bot]