dd-trace-java icon indicating copy to clipboard operation
dd-trace-java copied to clipboard

Published 20 hours ago • verified publisher icon DataDog

API Security sampling

Open ValentinZakharov opened this issue 1 year ago • 1 comments

What Does This Do

Introduced API Security sampling mechanism

Motivation

Additional Notes

Jira ticket: [PROJ-IDENT]

ValentinZakharov avatar Apr 16 '24 21:04 ValentinZakharov

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master vzakharov/api_sec_sampling
git_commit_date 1713861708 1713913003
git_commit_sha f34dfb52a1 1cf51e2d2c
release_version 1.34.0-SNAPSHOT~f34dfb52a1 1.33.0-SNAPSHOT~1cf51e2d2c
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1713915653 1713915653
ci_job_id 495028221 495028221
ci_pipeline_id 32766847 32766847
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module Agent Agent
parent None None
variant iast iast

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 50 metrics, 12 unstable metrics.

scenario Δ mean execution_time candidate mean execution_time baseline mean execution_time
scenario:startup:petclinic:profiling:Remote Config better
[-67.396µs; -17.973µs] or [-9.373%; -2.500%]		 676.358µs 719.043µs
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.33.0-SNAPSHOT~1cf51e2d2c, baseline=1.34.0-SNAPSHOT~f34dfb52a1

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.075 s) : 0, 1075443
Total [baseline] (8.545 s) : 0, 8545079
Agent [candidate] (1.083 s) : 0, 1082951
Total [candidate] (8.578 s) : 0, 8578386
section iast
Agent [baseline] (1.198 s) : 0, 1197709
Total [baseline] (9.007 s) : 0, 9006846
Agent [candidate] (1.199 s) : 0, 1198921
Total [candidate] (8.998 s) : 0, 8998033
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.197 s) : 0, 1197137
Total [baseline] (8.978 s) : 0, 8977894
Agent [candidate] (1.202 s) : 0, 1201630
Total [candidate] (8.974 s) : 0, 8974189
section iast_TELEMETRY_OFF
Agent [baseline] (1.196 s) : 0, 1196258
Total [baseline] (8.998 s) : 0, 8997555
Agent [candidate] (1.204 s) : 0, 1204267
Total [candidate] (9.011 s) : 0, 9010679
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.075 s -
Agent iast 1.198 s 122.266 ms (11.4%)
Agent iast_HARDCODED_SECRET_DISABLED 1.197 s 121.694 ms (11.3%)
Agent iast_TELEMETRY_OFF 1.196 s 120.815 ms (11.2%)
Total tracing 8.545 s -
Total iast 9.007 s 461.767 ms (5.4%)
Total iast_HARDCODED_SECRET_DISABLED 8.978 s 432.815 ms (5.1%)
Total iast_TELEMETRY_OFF 8.998 s 452.476 ms (5.3%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.083 s -
Agent iast 1.199 s 115.97 ms (10.7%)
Agent iast_HARDCODED_SECRET_DISABLED 1.202 s 118.679 ms (11.0%)
Agent iast_TELEMETRY_OFF 1.204 s 121.317 ms (11.2%)
Total tracing 8.578 s -
Total iast 8.998 s 419.648 ms (4.9%)
Total iast_HARDCODED_SECRET_DISABLED 8.974 s 395.804 ms (4.6%)
Total iast_TELEMETRY_OFF 9.011 s 432.293 ms (5.0%) 
gantt
    title insecure-bank - break down per module: candidate=1.33.0-SNAPSHOT~1cf51e2d2c, baseline=1.34.0-SNAPSHOT~f34dfb52a1

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (673.321 ms) : 0, 673321
BytebuddyAgent [candidate] (678.281 ms) : 0, 678281
GlobalTracer [baseline] (310.021 ms) : 0, 310021
GlobalTracer [candidate] (312.13 ms) : 0, 312130
AppSec [baseline] (49.477 ms) : 0, 49477
AppSec [candidate] (49.592 ms) : 0, 49592
Remote Config [baseline] (655.995 µs) : 0, 656
Remote Config [candidate] (666.146 µs) : 0, 666
Telemetry [baseline] (7.579 ms) : 0, 7579
Telemetry [candidate] (7.681 ms) : 0, 7681
section iast
BytebuddyAgent [baseline] (793.964 ms) : 0, 793964
BytebuddyAgent [candidate] (795.018 ms) : 0, 795018
GlobalTracer [baseline] (287.989 ms) : 0, 287989
GlobalTracer [candidate] (288.458 ms) : 0, 288458
AppSec [baseline] (50.258 ms) : 0, 50258
AppSec [candidate] (50.137 ms) : 0, 50137
Remote Config [baseline] (579.191 µs) : 0, 579
Remote Config [candidate] (573.342 µs) : 0, 573
Telemetry [baseline] (8.107 ms) : 0, 8107
Telemetry [candidate] (6.632 ms) : 0, 6632
IAST [baseline] (22.582 ms) : 0, 22582
IAST [candidate] (23.653 ms) : 0, 23653
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (793.507 ms) : 0, 793507
BytebuddyAgent [candidate] (796.203 ms) : 0, 796203
GlobalTracer [baseline] (287.413 ms) : 0, 287413
GlobalTracer [candidate] (289.123 ms) : 0, 289123
AppSec [baseline] (51.556 ms) : 0, 51556
AppSec [candidate] (49.077 ms) : 0, 49077
Remote Config [baseline] (563.449 µs) : 0, 563
Remote Config [candidate] (583.68 µs) : 0, 584
Telemetry [baseline] (6.548 ms) : 0, 6548
Telemetry [candidate] (6.571 ms) : 0, 6571
IAST [baseline] (23.185 ms) : 0, 23185
IAST [candidate] (25.51 ms) : 0, 25510
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (792.551 ms) : 0, 792551
BytebuddyAgent [candidate] (797.729 ms) : 0, 797729
GlobalTracer [baseline] (288.425 ms) : 0, 288425
GlobalTracer [candidate] (290.589 ms) : 0, 290589
AppSec [baseline] (48.797 ms) : 0, 48797
AppSec [candidate] (47.75 ms) : 0, 47750
Remote Config [baseline] (581.322 µs) : 0, 581
Remote Config [candidate] (576.048 µs) : 0, 576
Telemetry [baseline] (8.808 ms) : 0, 8808
Telemetry [candidate] (7.264 ms) : 0, 7264
IAST [baseline] (22.789 ms) : 0, 22789
IAST [candidate] (25.777 ms) : 0, 25777
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.33.0-SNAPSHOT~1cf51e2d2c, baseline=1.34.0-SNAPSHOT~f34dfb52a1

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.092 s) : 0, 1092001
Total [baseline] (10.498 s) : 0, 10497704
Agent [candidate] (1.075 s) : 0, 1074608
Total [candidate] (10.26 s) : 0, 10260141
section appsec
Agent [baseline] (1.198 s) : 0, 1198435
Total [baseline] (10.427 s) : 0, 10426703
Agent [candidate] (1.205 s) : 0, 1204981
Total [candidate] (10.557 s) : 0, 10556580
section iast
Agent [baseline] (1.199 s) : 0, 1199112
Total [baseline] (10.706 s) : 0, 10706410
Agent [candidate] (1.198 s) : 0, 1197909
Total [candidate] (10.698 s) : 0, 10697882
section profiling
Agent [baseline] (1.276 s) : 0, 1275887
Total [baseline] (10.642 s) : 0, 10641717
Agent [candidate] (1.264 s) : 0, 1264305
Total [candidate] (10.59 s) : 0, 10590490
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.092 s -
Agent appsec 1.198 s 106.434 ms (9.7%)
Agent iast 1.199 s 107.11 ms (9.8%)
Agent profiling 1.276 s 183.886 ms (16.8%)
Total tracing 10.498 s -
Total appsec 10.427 s -71.001 ms (-0.7%)
Total iast 10.706 s 208.706 ms (2.0%)
Total profiling 10.642 s 144.013 ms (1.4%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.075 s -
Agent appsec 1.205 s 130.373 ms (12.1%)
Agent iast 1.198 s 123.3 ms (11.5%)
Agent profiling 1.264 s 189.697 ms (17.7%)
Total tracing 10.26 s -
Total appsec 10.557 s 296.439 ms (2.9%)
Total iast 10.698 s 437.741 ms (4.3%)
Total profiling 10.59 s 330.349 ms (3.2%) 
gantt
    title petclinic - break down per module: candidate=1.33.0-SNAPSHOT~1cf51e2d2c, baseline=1.34.0-SNAPSHOT~f34dfb52a1

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (683.927 ms) : 0, 683927
BytebuddyAgent [candidate] (673.467 ms) : 0, 673467
GlobalTracer [baseline] (314.401 ms) : 0, 314401
GlobalTracer [candidate] (309.624 ms) : 0, 309624
AppSec [baseline] (50.414 ms) : 0, 50414
AppSec [candidate] (48.879 ms) : 0, 48879
Remote Config [baseline] (682.211 µs) : 0, 682
Remote Config [candidate] (653.935 µs) : 0, 654
Telemetry [baseline] (7.714 ms) : 0, 7714
Telemetry [candidate] (7.575 ms) : 0, 7575
section appsec
BytebuddyAgent [baseline] (694.195 ms) : 0, 694195
BytebuddyAgent [candidate] (700.269 ms) : 0, 700269
GlobalTracer [baseline] (290.607 ms) : 0, 290607
GlobalTracer [candidate] (294.066 ms) : 0, 294066
AppSec [baseline] (149.529 ms) : 0, 149529
AppSec [candidate] (148.794 ms) : 0, 148794
IAST [baseline] (19.198 ms) : 0, 19198
IAST [candidate] (19.094 ms) : 0, 19094
Remote Config [baseline] (606.369 µs) : 0, 606
Remote Config [candidate] (615.704 µs) : 0, 616
Telemetry [baseline] (9.965 ms) : 0, 9965
Telemetry [candidate] (7.473 ms) : 0, 7473
section iast
BytebuddyAgent [baseline] (794.244 ms) : 0, 794244
BytebuddyAgent [candidate] (794.011 ms) : 0, 794011
GlobalTracer [baseline] (288.404 ms) : 0, 288404
GlobalTracer [candidate] (288.139 ms) : 0, 288139
AppSec [baseline] (50.154 ms) : 0, 50154
AppSec [candidate] (49.803 ms) : 0, 49803
IAST [baseline] (23.339 ms) : 0, 23339
IAST [candidate] (24.492 ms) : 0, 24492
Remote Config [baseline] (571.624 µs) : 0, 572
Remote Config [candidate] (563.887 µs) : 0, 564
Telemetry [baseline] (8.144 ms) : 0, 8144
Telemetry [candidate] (6.567 ms) : 0, 6567
section profiling
BytebuddyAgent [baseline] (682.572 ms) : 0, 682572
BytebuddyAgent [candidate] (675.942 ms) : 0, 675942
GlobalTracer [baseline] (381.756 ms) : 0, 381756
GlobalTracer [candidate] (379.104 ms) : 0, 379104
AppSec [baseline] (50.427 ms) : 0, 50427
AppSec [candidate] (49.495 ms) : 0, 49495
Remote Config [baseline] (719.043 µs) : 0, 719
Remote Config [candidate] (676.358 µs) : 0, 676
Telemetry [baseline] (7.502 ms) : 0, 7502
Telemetry [candidate] (7.401 ms) : 0, 7401
ProfilingAgent [baseline] (96.113 ms) : 0, 96113
ProfilingAgent [candidate] (95.591 ms) : 0, 95591
Profiling [baseline] (96.136 ms) : 0, 96136
Profiling [candidate] (95.615 ms) : 0, 95615

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2024-04-23T23:10:40 2024-04-23T23:17:28
git_branch master vzakharov/api_sec_sampling
git_commit_date 1713861708 1713913003
git_commit_sha f34dfb52a1 1cf51e2d2c
release_version 1.34.0-SNAPSHOT~f34dfb52a1 1.33.0-SNAPSHOT~1cf51e2d2c
start_time 2024-04-23T23:10:27 2024-04-23T23:17:15
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1713914592 1713914592
ci_job_id 495028222 495028222
ci_pipeline_id 32766847 32766847
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 17 unstable metrics.

Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.33.0-SNAPSHOT~1cf51e2d2c, baseline=1.34.0-SNAPSHOT~f34dfb52a1
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.331 ms) : 1311, 1350
.   : milestone, 1331,
appsec (1.715 ms) : 1690, 1739
.   : milestone, 1715,
appsec_no_iast (1.727 ms) : 1702, 1751
.   : milestone, 1727,
iast (1.501 ms) : 1479, 1523
.   : milestone, 1501,
profiling (1.486 ms) : 1462, 1510
.   : milestone, 1486,
tracing (1.483 ms) : 1459, 1507
.   : milestone, 1483,
section candidate
no_agent (1.336 ms) : 1317, 1355
.   : milestone, 1336,
appsec (1.722 ms) : 1697, 1746
.   : milestone, 1722,
appsec_no_iast (1.722 ms) : 1698, 1745
.   : milestone, 1722,
iast (1.479 ms) : 1456, 1502
.   : milestone, 1479,
profiling (1.524 ms) : 1498, 1551
.   : milestone, 1524,
tracing (1.487 ms) : 1463, 1511
.   : milestone, 1487,
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.331 ms [1.311 ms, 1.35 ms] -
appsec 1.715 ms [1.69 ms, 1.739 ms] 383.799 µs (28.8%)
appsec_no_iast 1.727 ms [1.702 ms, 1.751 ms] 395.806 µs (29.7%)
iast 1.501 ms [1.479 ms, 1.523 ms] 170.358 µs (12.8%)
profiling 1.486 ms [1.462 ms, 1.51 ms] 155.384 µs (11.7%)
tracing 1.483 ms [1.459 ms, 1.507 ms] 152.101 µs (11.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 1.336 ms [1.317 ms, 1.355 ms] -
appsec 1.722 ms [1.697 ms, 1.746 ms] 385.371 µs (28.8%)
appsec_no_iast 1.722 ms [1.698 ms, 1.745 ms] 385.389 µs (28.8%)
iast 1.479 ms [1.456 ms, 1.502 ms] 142.781 µs (10.7%)
profiling 1.524 ms [1.498 ms, 1.551 ms] 188.23 µs (14.1%)
tracing 1.487 ms [1.463 ms, 1.511 ms] 151.048 µs (11.3%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.33.0-SNAPSHOT~1cf51e2d2c, baseline=1.34.0-SNAPSHOT~f34dfb52a1
    dateFormat X
    axisFormat %s
section baseline
no_agent (364.387 µs) : 344, 385
.   : milestone, 364,
iast (474.487 µs) : 453, 496
.   : milestone, 474,
iast_FULL (535.456 µs) : 515, 556
.   : milestone, 535,
iast_GLOBAL (495.907 µs) : 474, 518
.   : milestone, 496,
iast_HARDCODED_SECRET_DISABLED (469.974 µs) : 448, 492
.   : milestone, 470,
iast_INACTIVE (450.148 µs) : 429, 471
.   : milestone, 450,
iast_TELEMETRY_OFF (469.62 µs) : 448, 492
.   : milestone, 470,
tracing (442.657 µs) : 422, 463
.   : milestone, 443,
section candidate
no_agent (368.789 µs) : 349, 388
.   : milestone, 369,
iast (477.915 µs) : 456, 499
.   : milestone, 478,
iast_FULL (538.443 µs) : 517, 559
.   : milestone, 538,
iast_GLOBAL (497.838 µs) : 477, 519
.   : milestone, 498,
iast_HARDCODED_SECRET_DISABLED (475.286 µs) : 453, 498
.   : milestone, 475,
iast_INACTIVE (446.095 µs) : 425, 467
.   : milestone, 446,
iast_TELEMETRY_OFF (466.673 µs) : 446, 487
.   : milestone, 467,
tracing (443.308 µs) : 422, 464
.   : milestone, 443,
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 364.387 µs [344.017 µs, 384.757 µs] -
iast 474.487 µs [453.213 µs, 495.761 µs] 110.101 µs (30.2%)
iast_FULL 535.456 µs [514.577 µs, 556.336 µs] 171.07 µs (46.9%)
iast_GLOBAL 495.907 µs [474.271 µs, 517.544 µs] 131.521 µs (36.1%)
iast_HARDCODED_SECRET_DISABLED 469.974 µs [448.301 µs, 491.647 µs] 105.587 µs (29.0%)
iast_INACTIVE 450.148 µs [429.128 µs, 471.169 µs] 85.762 µs (23.5%)
iast_TELEMETRY_OFF 469.62 µs [447.566 µs, 491.674 µs] 105.233 µs (28.9%)
tracing 442.657 µs [422.059 µs, 463.255 µs] 78.27 µs (21.5%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 368.789 µs [349.257 µs, 388.322 µs] -
iast 477.915 µs [456.363 µs, 499.466 µs] 109.126 µs (29.6%)
iast_FULL 538.443 µs [517.436 µs, 559.45 µs] 169.654 µs (46.0%)
iast_GLOBAL 497.838 µs [476.768 µs, 518.907 µs] 129.048 µs (35.0%)
iast_HARDCODED_SECRET_DISABLED 475.286 µs [453.01 µs, 497.561 µs] 106.497 µs (28.9%)
iast_INACTIVE 446.095 µs [425.317 µs, 466.872 µs] 77.305 µs (21.0%)
iast_TELEMETRY_OFF 466.673 µs [445.977 µs, 487.368 µs] 97.884 µs (26.5%)
tracing 443.308 µs [422.321 µs, 464.295 µs] 74.519 µs (20.2%)

Dacapo

pr-commenter[bot] avatar Apr 16 '24 22:04 pr-commenter[bot]

This pull request has been marked as stale because it has not had activity over the past quarter. It will be closed in 7 days if no further activity occurs. Feel free to reopen the PR if you are still working on it.

github-actions[bot] avatar Nov 01 '24 03:11 github-actions[bot]

This pull request has been closed because it has not had activity over the past quarter. Feel free to reopen the PR if you are still working on it.

github-actions[bot] avatar Dec 01 '24 03:12 github-actions[bot]