dd-trace-java icon indicating copy to clipboard operation
dd-trace-java copied to clipboard

Published 20 hours ago • verified publisher icon DataDog

DRAFT

Open jpbempel opened this issue 1 year ago • 2 comments

What Does This Do

Motivation

Additional Notes

Jira ticket: [PROJ-IDENT]

jpbempel avatar Jul 16 '24 10:07 jpbempel

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master jpbempel/fix-benchmarks
git_commit_date 1721119353 1721123294
git_commit_sha 2efac1aa79 ed0bab4a69
release_version 1.38.0-SNAPSHOT~2efac1aa79 1.38.0-SNAPSHOT~ed0bab4a69
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1721125823 1721125823
ci_job_id 573448245 573448245
ci_pipeline_id 39248793 39248793
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 53 metrics, 10 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.38.0-SNAPSHOT~ed0bab4a69, baseline=1.38.0-SNAPSHOT~2efac1aa79

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.064 s) : 0, 1064077
Total [baseline] (10.271 s) : 0, 10271089
Agent [candidate] (1.064 s) : 0, 1064489
Total [candidate] (10.284 s) : 0, 10283702
section appsec
Agent [baseline] (1.183 s) : 0, 1182706
Total [baseline] (10.532 s) : 0, 10532140
Agent [candidate] (1.187 s) : 0, 1187208
Total [candidate] (10.549 s) : 0, 10549046
section iast
Agent [baseline] (1.181 s) : 0, 1181169
Total [baseline] (10.706 s) : 0, 10705976
Agent [candidate] (1.175 s) : 0, 1174668
Total [candidate] (10.736 s) : 0, 10735622
section profiling
Agent [baseline] (1.265 s) : 0, 1265498
Total [baseline] (10.598 s) : 0, 10597685
Agent [candidate] (1.283 s) : 0, 1282882
Total [candidate] (10.668 s) : 0, 10668068
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.064 s -
Agent appsec 1.183 s 118.629 ms (11.1%)
Agent iast 1.181 s 117.092 ms (11.0%)
Agent profiling 1.265 s 201.421 ms (18.9%)
Total tracing 10.271 s -
Total appsec 10.532 s 261.051 ms (2.5%)
Total iast 10.706 s 434.888 ms (4.2%)
Total profiling 10.598 s 326.597 ms (3.2%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.064 s -
Agent appsec 1.187 s 122.719 ms (11.5%)
Agent iast 1.175 s 110.179 ms (10.4%)
Agent profiling 1.283 s 218.393 ms (20.5%)
Total tracing 10.284 s -
Total appsec 10.549 s 265.344 ms (2.6%)
Total iast 10.736 s 451.92 ms (4.4%)
Total profiling 10.668 s 384.366 ms (3.7%) 
gantt
    title petclinic - break down per module: candidate=1.38.0-SNAPSHOT~ed0bab4a69, baseline=1.38.0-SNAPSHOT~2efac1aa79

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (666.458 ms) : 0, 666458
BytebuddyAgent [candidate] (666.308 ms) : 0, 666308
GlobalTracer [baseline] (304.83 ms) : 0, 304830
GlobalTracer [candidate] (305.158 ms) : 0, 305158
AppSec [baseline] (49.887 ms) : 0, 49887
AppSec [candidate] (49.977 ms) : 0, 49977
Remote Config [baseline] (665.711 µs) : 0, 666
Remote Config [candidate] (666.32 µs) : 0, 666
Telemetry [baseline] (7.604 ms) : 0, 7604
Telemetry [candidate] (7.667 ms) : 0, 7667
section appsec
BytebuddyAgent [baseline] (676.035 ms) : 0, 676035
BytebuddyAgent [candidate] (678.411 ms) : 0, 678411
GlobalTracer [baseline] (298.887 ms) : 0, 298887
GlobalTracer [candidate] (299.542 ms) : 0, 299542
AppSec [baseline] (153.497 ms) : 0, 153497
AppSec [candidate] (154.087 ms) : 0, 154087
Remote Config [baseline] (615.678 µs) : 0, 616
Remote Config [candidate] (619.309 µs) : 0, 619
Telemetry [baseline] (8.237 ms) : 0, 8237
Telemetry [candidate] (8.914 ms) : 0, 8914
IAST [baseline] (21.525 ms) : 0, 21525
IAST [candidate] (21.521 ms) : 0, 21521
section iast
BytebuddyAgent [baseline] (786.257 ms) : 0, 786257
BytebuddyAgent [candidate] (781.888 ms) : 0, 781888
GlobalTracer [baseline] (297.699 ms) : 0, 297699
GlobalTracer [candidate] (296.361 ms) : 0, 296361
AppSec [baseline] (48.741 ms) : 0, 48741
AppSec [candidate] (47.252 ms) : 0, 47252
Remote Config [baseline] (568.09 µs) : 0, 568
Remote Config [candidate] (569.203 µs) : 0, 569
Telemetry [baseline] (7.773 ms) : 0, 7773
Telemetry [candidate] (6.919 ms) : 0, 6919
IAST [baseline] (26.459 ms) : 0, 26459
IAST [candidate] (28.08 ms) : 0, 28080
section profiling
BytebuddyAgent [baseline] (663.822 ms) : 0, 663822
BytebuddyAgent [candidate] (673.224 ms) : 0, 673224
GlobalTracer [baseline] (388.396 ms) : 0, 388396
GlobalTracer [candidate] (392.719 ms) : 0, 392719
AppSec [baseline] (51.882 ms) : 0, 51882
AppSec [candidate] (52.273 ms) : 0, 52273
Remote Config [baseline] (671.256 µs) : 0, 671
Remote Config [candidate] (667.556 µs) : 0, 668
Telemetry [baseline] (7.331 ms) : 0, 7331
Telemetry [candidate] (7.435 ms) : 0, 7435
ProfilingAgent [baseline] (96.134 ms) : 0, 96134
ProfilingAgent [candidate] (98.417 ms) : 0, 98417
Profiling [baseline] (96.161 ms) : 0, 96161
Profiling [candidate] (98.442 ms) : 0, 98442
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.38.0-SNAPSHOT~ed0bab4a69, baseline=1.38.0-SNAPSHOT~2efac1aa79

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.073 s) : 0, 1072589
Total [baseline] (8.547 s) : 0, 8547237
Agent [candidate] (1.067 s) : 0, 1067310
Total [candidate] (8.52 s) : 0, 8519684
section iast
Agent [baseline] (1.174 s) : 0, 1174082
Total [baseline] (8.967 s) : 0, 8967457
Agent [candidate] (1.172 s) : 0, 1172002
Total [candidate] (8.957 s) : 0, 8957182
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.183 s) : 0, 1182692
Total [baseline] (8.952 s) : 0, 8951952
Agent [candidate] (1.173 s) : 0, 1172987
Total [candidate] (9.002 s) : 0, 9001660
section iast_TELEMETRY_OFF
Agent [baseline] (1.167 s) : 0, 1166566
Total [baseline] (8.984 s) : 0, 8984040
Agent [candidate] (1.174 s) : 0, 1173575
Total [candidate] (8.942 s) : 0, 8942153
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.073 s -
Agent iast 1.174 s 101.493 ms (9.5%)
Agent iast_HARDCODED_SECRET_DISABLED 1.183 s 110.103 ms (10.3%)
Agent iast_TELEMETRY_OFF 1.167 s 93.977 ms (8.8%)
Total tracing 8.547 s -
Total iast 8.967 s 420.22 ms (4.9%)
Total iast_HARDCODED_SECRET_DISABLED 8.952 s 404.715 ms (4.7%)
Total iast_TELEMETRY_OFF 8.984 s 436.803 ms (5.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.067 s -
Agent iast 1.172 s 104.692 ms (9.8%)
Agent iast_HARDCODED_SECRET_DISABLED 1.173 s 105.677 ms (9.9%)
Agent iast_TELEMETRY_OFF 1.174 s 106.265 ms (10.0%)
Total tracing 8.52 s -
Total iast 8.957 s 437.499 ms (5.1%)
Total iast_HARDCODED_SECRET_DISABLED 9.002 s 481.976 ms (5.7%)
Total iast_TELEMETRY_OFF 8.942 s 422.469 ms (5.0%) 
gantt
    title insecure-bank - break down per module: candidate=1.38.0-SNAPSHOT~ed0bab4a69, baseline=1.38.0-SNAPSHOT~2efac1aa79

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (671.968 ms) : 0, 671968
BytebuddyAgent [candidate] (668.078 ms) : 0, 668078
GlobalTracer [baseline] (307.002 ms) : 0, 307002
GlobalTracer [candidate] (306.24 ms) : 0, 306240
AppSec [baseline] (50.412 ms) : 0, 50412
AppSec [candidate] (50.125 ms) : 0, 50125
Remote Config [baseline] (679.24 µs) : 0, 679
Remote Config [candidate] (679.349 µs) : 0, 679
Telemetry [baseline] (7.634 ms) : 0, 7634
Telemetry [candidate] (7.57 ms) : 0, 7570
section iast
BytebuddyAgent [baseline] (782.958 ms) : 0, 782958
BytebuddyAgent [candidate] (780.58 ms) : 0, 780580
GlobalTracer [baseline] (295.984 ms) : 0, 295984
GlobalTracer [candidate] (295.288 ms) : 0, 295288
AppSec [baseline] (49.068 ms) : 0, 49068
AppSec [candidate] (49.613 ms) : 0, 49613
Remote Config [baseline] (574.656 µs) : 0, 575
Remote Config [candidate] (569.272 µs) : 0, 569
Telemetry [baseline] (7.023 ms) : 0, 7023
Telemetry [candidate] (6.995 ms) : 0, 6995
IAST [baseline] (24.919 ms) : 0, 24919
IAST [candidate] (25.429 ms) : 0, 25429
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (786.852 ms) : 0, 786852
BytebuddyAgent [candidate] (780.638 ms) : 0, 780638
GlobalTracer [baseline] (297.486 ms) : 0, 297486
GlobalTracer [candidate] (294.96 ms) : 0, 294960
AppSec [baseline] (50.12 ms) : 0, 50120
AppSec [candidate] (49.714 ms) : 0, 49714
Remote Config [baseline] (584.126 µs) : 0, 584
Remote Config [candidate] (578.811 µs) : 0, 579
Telemetry [baseline] (7.079 ms) : 0, 7079
Telemetry [candidate] (7.796 ms) : 0, 7796
IAST [baseline] (26.847 ms) : 0, 26847
IAST [candidate] (25.66 ms) : 0, 25660
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (776.95 ms) : 0, 776950
BytebuddyAgent [candidate] (782.785 ms) : 0, 782785
GlobalTracer [baseline] (294.626 ms) : 0, 294626
GlobalTracer [candidate] (295.119 ms) : 0, 295119
AppSec [baseline] (47.218 ms) : 0, 47218
AppSec [candidate] (47.086 ms) : 0, 47086
Remote Config [baseline] (564.373 µs) : 0, 564
Remote Config [candidate] (562.111 µs) : 0, 562
Telemetry [baseline] (6.807 ms) : 0, 6807
Telemetry [candidate] (6.811 ms) : 0, 6811
IAST [baseline] (26.855 ms) : 0, 26855
IAST [candidate] (27.502 ms) : 0, 27502

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-07-16T10:01:11 2024-07-16T10:08:01
git_branch master jpbempel/fix-benchmarks
git_commit_date 1721119353 1721123294
git_commit_sha 2efac1aa79 ed0bab4a69
release_version 1.38.0-SNAPSHOT~2efac1aa79 1.38.0-SNAPSHOT~ed0bab4a69
start_time 2024-07-16T10:00:58 2024-07-16T10:07:48
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1721124826 1721124826
ci_job_id 573448246 573448246
ci_pipeline_id 39248793 39248793
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.38.0-SNAPSHOT~ed0bab4a69, baseline=1.38.0-SNAPSHOT~2efac1aa79
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.335 ms) : 1316, 1353
.   : milestone, 1335,
appsec (1.72 ms) : 1696, 1743
.   : milestone, 1720,
appsec_no_iast (1.725 ms) : 1701, 1748
.   : milestone, 1725,
iast (1.487 ms) : 1465, 1509
.   : milestone, 1487,
profiling (1.542 ms) : 1516, 1569
.   : milestone, 1542,
tracing (1.473 ms) : 1449, 1496
.   : milestone, 1473,
section candidate
no_agent (1.335 ms) : 1316, 1353
.   : milestone, 1335,
appsec (1.72 ms) : 1696, 1744
.   : milestone, 1720,
appsec_no_iast (1.712 ms) : 1686, 1737
.   : milestone, 1712,
iast (1.472 ms) : 1449, 1495
.   : milestone, 1472,
profiling (1.55 ms) : 1524, 1576
.   : milestone, 1550,
tracing (1.463 ms) : 1439, 1487
.   : milestone, 1463,
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.335 ms [1.316 ms, 1.353 ms] -
appsec 1.72 ms [1.696 ms, 1.743 ms] 384.996 µs (28.8%)
appsec_no_iast 1.725 ms [1.701 ms, 1.748 ms] 390.103 µs (29.2%)
iast 1.487 ms [1.465 ms, 1.509 ms] 152.248 µs (11.4%)
profiling 1.542 ms [1.516 ms, 1.569 ms] 207.493 µs (15.5%)
tracing 1.473 ms [1.449 ms, 1.496 ms] 137.939 µs (10.3%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.335 ms [1.316 ms, 1.353 ms] -
appsec 1.72 ms [1.696 ms, 1.744 ms] 385.532 µs (28.9%)
appsec_no_iast 1.712 ms [1.686 ms, 1.737 ms] 377.358 µs (28.3%)
iast 1.472 ms [1.449 ms, 1.495 ms] 137.084 µs (10.3%)
profiling 1.55 ms [1.524 ms, 1.576 ms] 215.483 µs (16.1%)
tracing 1.463 ms [1.439 ms, 1.487 ms] 128.403 µs (9.6%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.38.0-SNAPSHOT~ed0bab4a69, baseline=1.38.0-SNAPSHOT~2efac1aa79
    dateFormat X
    axisFormat %s
section baseline
no_agent (366.542 µs) : 347, 386
.   : milestone, 367,
iast (478.842 µs) : 458, 500
.   : milestone, 479,
iast_FULL (550.724 µs) : 530, 572
.   : milestone, 551,
iast_GLOBAL (498.933 µs) : 478, 520
.   : milestone, 499,
iast_HARDCODED_SECRET_DISABLED (482.183 µs) : 461, 504
.   : milestone, 482,
iast_INACTIVE (454.611 µs) : 433, 476
.   : milestone, 455,
iast_TELEMETRY_OFF (463.29 µs) : 442, 484
.   : milestone, 463,
tracing (444.218 µs) : 424, 465
.   : milestone, 444,
section candidate
no_agent (364.335 µs) : 344, 385
.   : milestone, 364,
iast (479.666 µs) : 458, 501
.   : milestone, 480,
iast_FULL (547.931 µs) : 527, 569
.   : milestone, 548,
iast_GLOBAL (500.973 µs) : 480, 522
.   : milestone, 501,
iast_HARDCODED_SECRET_DISABLED (472.757 µs) : 452, 493
.   : milestone, 473,
iast_INACTIVE (450.49 µs) : 430, 471
.   : milestone, 450,
iast_TELEMETRY_OFF (466.052 µs) : 445, 487
.   : milestone, 466,
tracing (447.464 µs) : 426, 469
.   : milestone, 447,
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 366.542 µs [347.182 µs, 385.902 µs] -
iast 478.842 µs [457.558 µs, 500.127 µs] 112.3 µs (30.6%)
iast_FULL 550.724 µs [529.836 µs, 571.612 µs] 184.182 µs (50.2%)
iast_GLOBAL 498.933 µs [478.091 µs, 519.775 µs] 132.391 µs (36.1%)
iast_HARDCODED_SECRET_DISABLED 482.183 µs [460.846 µs, 503.521 µs] 115.641 µs (31.5%)
iast_INACTIVE 454.611 µs [432.77 µs, 476.453 µs] 88.07 µs (24.0%)
iast_TELEMETRY_OFF 463.29 µs [442.426 µs, 484.154 µs] 96.748 µs (26.4%)
tracing 444.218 µs [423.723 µs, 464.714 µs] 77.677 µs (21.2%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 364.335 µs [343.533 µs, 385.137 µs] -
iast 479.666 µs [458.474 µs, 500.857 µs] 115.331 µs (31.7%)
iast_FULL 547.931 µs [527.003 µs, 568.859 µs] 183.596 µs (50.4%)
iast_GLOBAL 500.973 µs [479.715 µs, 522.232 µs] 136.638 µs (37.5%)
iast_HARDCODED_SECRET_DISABLED 472.757 µs [452.201 µs, 493.312 µs] 108.422 µs (29.8%)
iast_INACTIVE 450.49 µs [429.972 µs, 471.008 µs] 86.155 µs (23.6%)
iast_TELEMETRY_OFF 466.052 µs [445.053 µs, 487.051 µs] 101.717 µs (27.9%)
tracing 447.464 µs [426.365 µs, 468.564 µs] 83.129 µs (22.8%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master jpbempel/fix-benchmarks
git_commit_date 1721119353 1721123294
git_commit_sha 2efac1aa79 ed0bab4a69
release_version 1.38.0-SNAPSHOT~2efac1aa79 1.38.0-SNAPSHOT~ed0bab4a69
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1721125368 1721125368
ci_job_id 573448247 573448247
ci_pipeline_id 39248793 39248793
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.38.0-SNAPSHOT~ed0bab4a69, baseline=1.38.0-SNAPSHOT~2efac1aa79
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.456 ms) : 1445, 1468
.   : milestone, 1456,
appsec (2.219 ms) : 2184, 2254
.   : milestone, 2219,
iast (1.974 ms) : 1932, 2017
.   : milestone, 1974,
iast_GLOBAL (2.008 ms) : 1966, 2051
.   : milestone, 2008,
profiling (1.843 ms) : 1809, 1878
.   : milestone, 1843,
tracing (1.83 ms) : 1798, 1862
.   : milestone, 1830,
section candidate
no_agent (1.457 ms) : 1446, 1469
.   : milestone, 1457,
appsec (2.21 ms) : 2175, 2245
.   : milestone, 2210,
iast (1.965 ms) : 1923, 2007
.   : milestone, 1965,
iast_GLOBAL (2.006 ms) : 1964, 2049
.   : milestone, 2006,
profiling (1.854 ms) : 1820, 1888
.   : milestone, 1854,
tracing (1.827 ms) : 1795, 1860
.   : milestone, 1827,
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.456 ms [1.445 ms, 1.468 ms] -
appsec 2.219 ms [2.184 ms, 2.254 ms] 762.8 µs (52.4%)
iast 1.974 ms [1.932 ms, 2.017 ms] 518.283 µs (35.6%)
iast_GLOBAL 2.008 ms [1.966 ms, 2.051 ms] 552.15 µs (37.9%)
profiling 1.843 ms [1.809 ms, 1.878 ms] 387.028 µs (26.6%)
tracing 1.83 ms [1.798 ms, 1.862 ms] 373.807 µs (25.7%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.457 ms [1.446 ms, 1.469 ms] -
appsec 2.21 ms [2.175 ms, 2.245 ms] 752.251 µs (51.6%)
iast 1.965 ms [1.923 ms, 2.007 ms] 507.712 µs (34.8%)
iast_GLOBAL 2.006 ms [1.964 ms, 2.049 ms] 549.092 µs (37.7%)
profiling 1.854 ms [1.82 ms, 1.888 ms] 396.433 µs (27.2%)
tracing 1.827 ms [1.795 ms, 1.86 ms] 370.027 µs (25.4%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.38.0-SNAPSHOT~ed0bab4a69, baseline=1.38.0-SNAPSHOT~2efac1aa79
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.965 s) : 14965000, 14965000
.   : milestone, 14965000,
appsec (15.282 s) : 15282000, 15282000
.   : milestone, 15282000,
iast (18.899 s) : 18899000, 18899000
.   : milestone, 18899000,
iast_GLOBAL (17.852 s) : 17852000, 17852000
.   : milestone, 17852000,
profiling (15.229 s) : 15229000, 15229000
.   : milestone, 15229000,
tracing (15.093 s) : 15093000, 15093000
.   : milestone, 15093000,
section candidate
no_agent (15.15 s) : 15150000, 15150000
.   : milestone, 15150000,
appsec (15.207 s) : 15207000, 15207000
.   : milestone, 15207000,
iast (18.513 s) : 18513000, 18513000
.   : milestone, 18513000,
iast_GLOBAL (17.783 s) : 17783000, 17783000
.   : milestone, 17783000,
profiling (15.319 s) : 15319000, 15319000
.   : milestone, 15319000,
tracing (15.06 s) : 15060000, 15060000
.   : milestone, 15060000,
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.965 s [14.965 s, 14.965 s] -
appsec 15.282 s [15.282 s, 15.282 s] 317.0 ms (2.1%)
iast 18.899 s [18.899 s, 18.899 s] 3.934 s (26.3%)
iast_GLOBAL 17.852 s [17.852 s, 17.852 s] 2.887 s (19.3%)
profiling 15.229 s [15.229 s, 15.229 s] 264.0 ms (1.8%)
tracing 15.093 s [15.093 s, 15.093 s] 128.0 ms (0.9%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.15 s [15.15 s, 15.15 s] -
appsec 15.207 s [15.207 s, 15.207 s] 57.0 ms (0.4%)
iast 18.513 s [18.513 s, 18.513 s] 3.363 s (22.2%)
iast_GLOBAL 17.783 s [17.783 s, 17.783 s] 2.633 s (17.4%)
profiling 15.319 s [15.319 s, 15.319 s] 169.0 ms (1.1%)
tracing 15.06 s [15.06 s, 15.06 s] -90.0 ms (-0.6%)

pr-commenter[bot] avatar Jul 16 '24 10:07 pr-commenter[bot]

Debugger benchmarks

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
ci_job_date 1721125360 1721125749
end_time 2024-07-16T10:23:55 2024-07-16T10:30:22
git_branch master jpbempel/fix-benchmarks
git_commit_sha 2efac1aa79 ed0bab4a69ddb555f0b4ec4b4ac1ca51f792eeea
start_time 2024-07-16T10:22:41 2024-07-16T10:29:10
See matching parameters
Baseline Candidate
ci_job_id 573494807 573494807
ci_pipeline_id 39248793 39248793
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
git_commit_date 1721123294 1721123294

Summary

Found 5 performance improvements and 0 performance regressions! Performance is the same for 4 metrics, 6 unstable metrics.

scenario Δ mean agg_http_req_duration_min Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p75 Δ mean agg_http_req_duration_p99 Δ mean throughput
scenario:loop better
[-712.186µs; -695.013µs] or [-6.596%; -6.437%]		 better
[-782.788µs; -704.811µs] or [-7.121%; -6.411%]		 better
[-779.491µs; -661.474µs] or [-7.045%; -5.979%]		 better
[-980.417µs; -602.828µs] or [-8.561%; -5.264%]		 better
[+5.716op/s; +7.340op/s] or [+6.345%; +8.148%]
See unchanged results
scenario Δ mean agg_http_req_duration_min Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p75 Δ mean agg_http_req_duration_p99 Δ mean throughput
scenario:noprobe unstable
[-60.949µs; +18.536µs] or [-22.209%; +6.754%]		 unstable
[-79.437µs; +24.418µs] or [-25.310%; +7.780%]		 unstable
[-93.074µs; +32.192µs] or [-28.268%; +9.777%]		 unstable
[-876.501µs; +306.380µs] or [-94.616%; +33.073%]		 same
scenario:basic same same same unstable
[-59.859µs; +27.295µs] or [-9.959%; +4.541%]		 unstable
[-125.669op/s; +260.622op/s] or [-4.901%; +10.164%]
Request duration reports for reports 
gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (313.859 µs) : 248, 380
.   : milestone, 314,
basic (298.292 µs) : 287, 309
.   : milestone, 298,
loop (10.993 ms) : 10947, 11040
.   : milestone, 10993,
section candidate
noprobe (286.35 µs) : 263, 310
.   : milestone, 286,
basic (291.464 µs) : 284, 299
.   : milestone, 291,
loop (10.249 ms) : 10227, 10272
.   : milestone, 10249,
  • baseline results
Scenario Request median duration [CI 0.99]
noprobe 313.859 µs [247.745 µs, 379.973 µs]
basic 298.292 µs [287.108 µs, 309.475 µs]
loop 10.993 ms [10.947 ms, 11.04 ms]
  • candidate results
Scenario Request median duration [CI 0.99]
noprobe 286.35 µs [263.189 µs, 309.511 µs]
basic 291.464 µs [284.342 µs, 298.586 µs]
loop 10.249 ms [10.227 ms, 10.272 ms]

pr-commenter[bot] avatar Jul 16 '24 10:07 pr-commenter[bot]

This pull request has been marked as stale because it has not had activity over the past quarter. It will be closed in 7 days if no further activity occurs. Feel free to reopen the PR if you are still working on it.

github-actions[bot] avatar Nov 01 '24 03:11 github-actions[bot]