What does this PR do?
Add a new Gitlab CI job that dogfoods the Datadog SCA product
Motivation
@DataDog/software-integrity-and-trust partners with @DataDog/static-analysis to dogfood their SCA product and secure Datadog's supply chain.
Additional Notes
Benchmarks
Startup
Parameters
|
Baseline |
Candidate |
| baseline_or_candidate |
baseline |
candidate |
| git_branch |
master |
harmon.herring/sint-1892-deploy-sca |
| git_commit_date |
1712162876 |
1712169257 |
| git_commit_sha |
396fccabad |
a83f4e6ffd |
| release_version |
1.32.0-SNAPSHOT~396fccabad |
1.32.0-SNAPSHOT~a83f4e6ffd |
See matching parameters
|
Baseline |
Candidate |
| application |
insecure-bank |
insecure-bank |
| ci_job_date |
1712172635 |
1712172635 |
| ci_job_id |
477259672 |
477259672 |
| ci_pipeline_id |
31389857 |
31389857 |
| cpu_model |
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz |
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz |
| module |
Agent |
Agent |
| parent |
None |
None |
| variant |
iast |
iast |
Summary
Found 1 performance improvements and 0 performance regressions! Performance is the same for 50 metrics, 12 unstable metrics.
| scenario |
Δ mean execution_time |
candidate mean execution_time |
baseline mean execution_time |
| scenario:startup:insecure-bank:tracing:GlobalTracer |
better
[-12.616ms; -9.912ms] or [-4.086%; -3.210%] |
297.473ms |
308.736ms |
Startup time reports for insecure-bank
gantt
title insecure-bank - global startup overhead: candidate=1.32.0-SNAPSHOT~a83f4e6ffd, baseline=1.32.0-SNAPSHOT~396fccabad
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.073 s) : 0, 1073097
Total [baseline] (8.546 s) : 0, 8546402
Agent [candidate] (1.074 s) : 0, 1074414
Total [candidate] (8.562 s) : 0, 8562476
section iast
Agent [baseline] (1.197 s) : 0, 1197492
Total [baseline] (9.05 s) : 0, 9049647
Agent [candidate] (1.198 s) : 0, 1197653
Total [candidate] (9.077 s) : 0, 9077170
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.199 s) : 0, 1199343
Total [baseline] (9.007 s) : 0, 9007147
Agent [candidate] (1.198 s) : 0, 1198318
Total [candidate] (9.047 s) : 0, 9047426
section iast_TELEMETRY_OFF
Agent [baseline] (1.196 s) : 0, 1195650
Total [baseline] (9.0 s) : 0, 9000451
Agent [candidate] (1.197 s) : 0, 1197121
Total [candidate] (9.084 s) : 0, 9083634
| Module |
Variant |
Duration |
Δ tracing |
| Agent |
tracing |
1.073 s |
- |
| Agent |
iast |
1.197 s |
124.395 ms (11.6%) |
| Agent |
iast_HARDCODED_SECRET_DISABLED |
1.199 s |
126.245 ms (11.8%) |
| Agent |
iast_TELEMETRY_OFF |
1.196 s |
122.553 ms (11.4%) |
| Total |
tracing |
8.546 s |
- |
| Total |
iast |
9.05 s |
503.244 ms (5.9%) |
| Total |
iast_HARDCODED_SECRET_DISABLED |
9.007 s |
460.744 ms (5.4%) |
| Total |
iast_TELEMETRY_OFF |
9.0 s |
454.048 ms (5.3%) |
| Module |
Variant |
Duration |
Δ tracing |
| Agent |
tracing |
1.074 s |
- |
| Agent |
iast |
1.198 s |
123.238 ms (11.5%) |
| Agent |
iast_HARDCODED_SECRET_DISABLED |
1.198 s |
123.904 ms (11.5%) |
| Agent |
iast_TELEMETRY_OFF |
1.197 s |
122.707 ms (11.4%) |
| Total |
tracing |
8.562 s |
- |
| Total |
iast |
9.077 s |
514.694 ms (6.0%) |
| Total |
iast_HARDCODED_SECRET_DISABLED |
9.047 s |
484.949 ms (5.7%) |
| Total |
iast_TELEMETRY_OFF |
9.084 s |
521.157 ms (6.1%) |
gantt
title insecure-bank - break down per module: candidate=1.32.0-SNAPSHOT~a83f4e6ffd, baseline=1.32.0-SNAPSHOT~396fccabad
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (672.382 ms) : 0, 672382
BytebuddyAgent [candidate] (684.774 ms) : 0, 684774
GlobalTracer [baseline] (308.736 ms) : 0, 308736
GlobalTracer [candidate] (297.473 ms) : 0, 297473
AppSec [baseline] (49.288 ms) : 0, 49288
AppSec [candidate] (49.393 ms) : 0, 49393
Remote Config [baseline] (661.629 µs) : 0, 662
Remote Config [candidate] (657.492 µs) : 0, 657
Telemetry [baseline] (7.648 ms) : 0, 7648
Telemetry [candidate] (7.701 ms) : 0, 7701
section iast
BytebuddyAgent [baseline] (792.862 ms) : 0, 792862
BytebuddyAgent [candidate] (793.191 ms) : 0, 793191
GlobalTracer [baseline] (288.061 ms) : 0, 288061
GlobalTracer [candidate] (288.285 ms) : 0, 288285
AppSec [baseline] (49.002 ms) : 0, 49002
AppSec [candidate] (49.524 ms) : 0, 49524
IAST [baseline] (26.047 ms) : 0, 26047
IAST [candidate] (24.393 ms) : 0, 24393
Remote Config [baseline] (590.03 µs) : 0, 590
Remote Config [candidate] (586.574 µs) : 0, 587
Telemetry [baseline] (6.617 ms) : 0, 6617
Telemetry [candidate] (7.305 ms) : 0, 7305
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (794.708 ms) : 0, 794708
BytebuddyAgent [candidate] (792.944 ms) : 0, 792944
GlobalTracer [baseline] (288.038 ms) : 0, 288038
GlobalTracer [candidate] (288.93 ms) : 0, 288930
AppSec [baseline] (50.282 ms) : 0, 50282
AppSec [candidate] (50.33 ms) : 0, 50330
IAST [baseline] (24.641 ms) : 0, 24641
IAST [candidate] (24.457 ms) : 0, 24457
Remote Config [baseline] (580.507 µs) : 0, 581
Remote Config [candidate] (583.426 µs) : 0, 583
Telemetry [baseline] (6.704 ms) : 0, 6704
Telemetry [candidate] (6.673 ms) : 0, 6673
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (791.648 ms) : 0, 791648
BytebuddyAgent [candidate] (791.557 ms) : 0, 791557
GlobalTracer [baseline] (288.458 ms) : 0, 288458
GlobalTracer [candidate] (289.538 ms) : 0, 289538
AppSec [baseline] (50.2 ms) : 0, 50200
AppSec [candidate] (48.951 ms) : 0, 48951
IAST [baseline] (23.755 ms) : 0, 23755
IAST [candidate] (24.724 ms) : 0, 24724
Remote Config [baseline] (577.599 µs) : 0, 578
Remote Config [candidate] (574.295 µs) : 0, 574
Telemetry [baseline] (6.568 ms) : 0, 6568
Telemetry [candidate] (7.338 ms) : 0, 7338
Startup time reports for petclinic
gantt
title petclinic - global startup overhead: candidate=1.32.0-SNAPSHOT~a83f4e6ffd, baseline=1.32.0-SNAPSHOT~396fccabad
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.074 s) : 0, 1074313
Total [baseline] (10.418 s) : 0, 10417780
Agent [candidate] (1.081 s) : 0, 1080882
Total [candidate] (10.375 s) : 0, 10375380
section appsec
Agent [baseline] (1.202 s) : 0, 1202472
Total [baseline] (10.467 s) : 0, 10467495
Agent [candidate] (1.202 s) : 0, 1201566
Total [candidate] (10.622 s) : 0, 10621954
section iast
Agent [baseline] (1.204 s) : 0, 1204283
Total [baseline] (10.823 s) : 0, 10823354
Agent [candidate] (1.214 s) : 0, 1213579
Total [candidate] (10.783 s) : 0, 10783190
section profiling
Agent [baseline] (1.269 s) : 0, 1268554
Total [baseline] (10.607 s) : 0, 10607262
Agent [candidate] (1.267 s) : 0, 1267453
Total [candidate] (10.651 s) : 0, 10650733
| Module |
Variant |
Duration |
Δ tracing |
| Agent |
tracing |
1.074 s |
- |
| Agent |
appsec |
1.202 s |
128.158 ms (11.9%) |
| Agent |
iast |
1.204 s |
129.97 ms (12.1%) |
| Agent |
profiling |
1.269 s |
194.24 ms (18.1%) |
| Total |
tracing |
10.418 s |
- |
| Total |
appsec |
10.467 s |
49.715 ms (0.5%) |
| Total |
iast |
10.823 s |
405.573 ms (3.9%) |
| Total |
profiling |
10.607 s |
189.482 ms (1.8%) |
| Module |
Variant |
Duration |
Δ tracing |
| Agent |
tracing |
1.081 s |
- |
| Agent |
appsec |
1.202 s |
120.684 ms (11.2%) |
| Agent |
iast |
1.214 s |
132.697 ms (12.3%) |
| Agent |
profiling |
1.267 s |
186.571 ms (17.3%) |
| Total |
tracing |
10.375 s |
- |
| Total |
appsec |
10.622 s |
246.574 ms (2.4%) |
| Total |
iast |
10.783 s |
407.81 ms (3.9%) |
| Total |
profiling |
10.651 s |
275.354 ms (2.7%) |
gantt
title petclinic - break down per module: candidate=1.32.0-SNAPSHOT~a83f4e6ffd, baseline=1.32.0-SNAPSHOT~396fccabad
dateFormat X
axisFormat %s
section tracing
BytebuddyAgent [baseline] (673.171 ms) : 0, 673171
BytebuddyAgent [candidate] (688.574 ms) : 0, 688574
GlobalTracer [baseline] (309.088 ms) : 0, 309088
GlobalTracer [candidate] (299.362 ms) : 0, 299362
AppSec [baseline] (49.412 ms) : 0, 49412
AppSec [candidate] (49.998 ms) : 0, 49998
Remote Config [baseline] (660.456 µs) : 0, 660
Remote Config [candidate] (664.37 µs) : 0, 664
Telemetry [baseline] (7.603 ms) : 0, 7603
Telemetry [candidate] (7.616 ms) : 0, 7616
section appsec
BytebuddyAgent [baseline] (699.318 ms) : 0, 699318
BytebuddyAgent [candidate] (697.948 ms) : 0, 697948
GlobalTracer [baseline] (292.799 ms) : 0, 292799
GlobalTracer [candidate] (293.074 ms) : 0, 293074
AppSec [baseline] (149.581 ms) : 0, 149581
AppSec [candidate] (149.76 ms) : 0, 149760
IAST [baseline] (18.772 ms) : 0, 18772
IAST [candidate] (18.816 ms) : 0, 18816
Remote Config [baseline] (605.976 µs) : 0, 606
Remote Config [candidate] (613.354 µs) : 0, 613
Telemetry [baseline] (6.801 ms) : 0, 6801
Telemetry [candidate] (6.815 ms) : 0, 6815
section iast
BytebuddyAgent [baseline] (799.14 ms) : 0, 799140
BytebuddyAgent [candidate] (805.299 ms) : 0, 805299
GlobalTracer [baseline] (288.993 ms) : 0, 288993
GlobalTracer [candidate] (291.06 ms) : 0, 291060
AppSec [baseline] (50.568 ms) : 0, 50568
AppSec [candidate] (50.167 ms) : 0, 50167
IAST [baseline] (22.219 ms) : 0, 22219
IAST [candidate] (24.26 ms) : 0, 24260
Remote Config [baseline] (596.428 µs) : 0, 596
Remote Config [candidate] (583.692 µs) : 0, 584
Telemetry [baseline] (8.137 ms) : 0, 8137
Telemetry [candidate] (7.363 ms) : 0, 7363
section profiling
BytebuddyAgent [baseline] (676.197 ms) : 0, 676197
BytebuddyAgent [candidate] (676.629 ms) : 0, 676629
GlobalTracer [baseline] (381.467 ms) : 0, 381467
GlobalTracer [candidate] (380.458 ms) : 0, 380458
AppSec [baseline] (50.362 ms) : 0, 50362
AppSec [candidate] (50.226 ms) : 0, 50226
Remote Config [baseline] (708.151 µs) : 0, 708
Remote Config [candidate] (704.778 µs) : 0, 705
Telemetry [baseline] (7.532 ms) : 0, 7532
Telemetry [candidate] (7.52 ms) : 0, 7520
ProfilingAgent [baseline] (96.148 ms) : 0, 96148
ProfilingAgent [candidate] (95.819 ms) : 0, 95819
Profiling [baseline] (96.172 ms) : 0, 96172
Profiling [candidate] (95.842 ms) : 0, 95842
Load
Parameters
|
Baseline |
Candidate |
| baseline_or_candidate |
baseline |
candidate |
| end_time |
2024-04-03T19:02:52 |
2024-04-03T19:24:51 |
| git_branch |
master |
harmon.herring/sint-1892-deploy-sca |
| git_commit_date |
1712162876 |
1712169257 |
| git_commit_sha |
396fccabad |
a83f4e6ffd |
| release_version |
1.32.0-SNAPSHOT~396fccabad |
1.32.0-SNAPSHOT~a83f4e6ffd |
| start_time |
2024-04-03T19:02:38 |
2024-04-03T19:24:38 |
See matching parameters
|
Baseline |
Candidate |
| application |
insecure-bank |
insecure-bank |
| ci_job_date |
1712172635 |
1712172635 |
| ci_job_id |
477259672 |
477259672 |
| ci_pipeline_id |
31389857 |
31389857 |
| cpu_model |
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz |
Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz |
| variant |
iast |
iast |
Summary
Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 16 unstable metrics.
Request duration reports for insecure-bank
gantt
title insecure-bank - request duration [CI 0.99] : candidate=1.32.0-SNAPSHOT~a83f4e6ffd, baseline=1.32.0-SNAPSHOT~396fccabad
dateFormat X
axisFormat %s
section baseline
no_agent (364.844 µs) : 345, 385
. : milestone, 365,
iast (480.005 µs) : 458, 502
. : milestone, 480,
iast_FULL (539.99 µs) : 519, 561
. : milestone, 540,
iast_GLOBAL (504.734 µs) : 482, 527
. : milestone, 505,
iast_HARDCODED_SECRET_DISABLED (473.958 µs) : 453, 495
. : milestone, 474,
iast_INACTIVE (449.551 µs) : 428, 471
. : milestone, 450,
iast_TELEMETRY_OFF (468.474 µs) : 448, 489
. : milestone, 468,
tracing (436.477 µs) : 416, 456
. : milestone, 436,
section candidate
no_agent (371.82 µs) : 352, 392
. : milestone, 372,
iast (471.642 µs) : 451, 492
. : milestone, 472,
iast_FULL (538.546 µs) : 518, 559
. : milestone, 539,
iast_GLOBAL (503.181 µs) : 481, 525
. : milestone, 503,
iast_HARDCODED_SECRET_DISABLED (475.397 µs) : 454, 496
. : milestone, 475,
iast_INACTIVE (446.477 µs) : 426, 467
. : milestone, 446,
iast_TELEMETRY_OFF (466.13 µs) : 446, 486
. : milestone, 466,
tracing (442.353 µs) : 422, 463
. : milestone, 442,
| Variant |
Request duration [CI 0.99] |
Δ no_agent |
| no_agent |
364.844 µs [345.134 µs, 384.553 µs] |
- |
| iast |
480.005 µs [458.425 µs, 501.585 µs] |
115.162 µs (31.6%) |
| iast_FULL |
539.99 µs [518.9 µs, 561.08 µs] |
175.147 µs (48.0%) |
| iast_GLOBAL |
504.734 µs [482.151 µs, 527.317 µs] |
139.891 µs (38.3%) |
| iast_HARDCODED_SECRET_DISABLED |
473.958 µs [452.978 µs, 494.938 µs] |
109.115 µs (29.9%) |
| iast_INACTIVE |
449.551 µs [428.042 µs, 471.059 µs] |
84.707 µs (23.2%) |
| iast_TELEMETRY_OFF |
468.474 µs [448.006 µs, 488.942 µs] |
103.63 µs (28.4%) |
| tracing |
436.477 µs [416.466 µs, 456.488 µs] |
71.634 µs (19.6%) |
| Variant |
Request duration [CI 0.99] |
Δ no_agent |
| no_agent |
371.82 µs [351.595 µs, 392.044 µs] |
- |
| iast |
471.642 µs [450.959 µs, 492.326 µs] |
99.823 µs (26.8%) |
| iast_FULL |
538.546 µs [517.765 µs, 559.327 µs] |
166.727 µs (44.8%) |
| iast_GLOBAL |
503.181 µs [481.412 µs, 524.949 µs] |
131.361 µs (35.3%) |
| iast_HARDCODED_SECRET_DISABLED |
475.397 µs [454.348 µs, 496.446 µs] |
103.578 µs (27.9%) |
| iast_INACTIVE |
446.477 µs [425.539 µs, 467.414 µs] |
74.657 µs (20.1%) |
| iast_TELEMETRY_OFF |
466.13 µs [445.967 µs, 486.292 µs] |
94.31 µs (25.4%) |
| tracing |
442.353 µs [421.882 µs, 462.823 µs] |
70.533 µs (19.0%) |
Request duration reports for petclinic
gantt
title petclinic - request duration [CI 0.99] : candidate=1.32.0-SNAPSHOT~a83f4e6ffd, baseline=1.32.0-SNAPSHOT~396fccabad
dateFormat X
axisFormat %s
section baseline
no_agent (1.344 ms) : 1325, 1364
. : milestone, 1344,
appsec (1.719 ms) : 1694, 1744
. : milestone, 1719,
appsec_no_iast (1.721 ms) : 1696, 1745
. : milestone, 1721,
iast (1.49 ms) : 1467, 1513
. : milestone, 1490,
profiling (1.504 ms) : 1479, 1530
. : milestone, 1504,
tracing (1.464 ms) : 1439, 1489
. : milestone, 1464,
section candidate
no_agent (1.345 ms) : 1326, 1365
. : milestone, 1345,
appsec (1.715 ms) : 1691, 1739
. : milestone, 1715,
appsec_no_iast (1.726 ms) : 1702, 1750
. : milestone, 1726,
iast (1.484 ms) : 1462, 1507
. : milestone, 1484,
profiling (1.507 ms) : 1484, 1531
. : milestone, 1507,
tracing (1.468 ms) : 1444, 1492
. : milestone, 1468,
| Variant |
Request duration [CI 0.99] |
Δ no_agent |
| no_agent |
1.344 ms [1.325 ms, 1.364 ms] |
- |
| appsec |
1.719 ms [1.694 ms, 1.744 ms] |
374.773 µs (27.9%) |
| appsec_no_iast |
1.721 ms [1.696 ms, 1.745 ms] |
376.317 µs (28.0%) |
| iast |
1.49 ms [1.467 ms, 1.513 ms] |
145.807 µs (10.8%) |
| profiling |
1.504 ms [1.479 ms, 1.53 ms] |
160.121 µs (11.9%) |
| tracing |
1.464 ms [1.439 ms, 1.489 ms] |
119.496 µs (8.9%) |
| Variant |
Request duration [CI 0.99] |
Δ no_agent |
| no_agent |
1.345 ms [1.326 ms, 1.365 ms] |
- |
| appsec |
1.715 ms [1.691 ms, 1.739 ms] |
369.656 µs (27.5%) |
| appsec_no_iast |
1.726 ms [1.702 ms, 1.75 ms] |
380.596 µs (28.3%) |
| iast |
1.484 ms [1.462 ms, 1.507 ms] |
138.894 µs (10.3%) |
| profiling |
1.507 ms [1.484 ms, 1.531 ms] |
161.997 µs (12.0%) |
| tracing |
1.468 ms [1.444 ms, 1.492 ms] |
122.353 µs (9.1%) |
dd-trace-java copied to clipboard
DataDog
![pr-commenter[bot] avatar](https://avatars.githubusercontent.com/u/365230?v=4 "Published by a pub.dev verified publisher"){kind=small}