Do not buffer pending traces for CI Visibility

[nikita-tkachenko-datadog]

What Does This Do

Refactors datadog.trace.core.PendingTrace:

• an abstract parent class TraceCollector is created
• logic that is not related to pending traces buffering is extracted into the parent class
• the code using PendingTrace is changed to use TraceCollector
• a sibling class StreamingTraceCollector is created, which writes spans as they finish
• the new streaming collector is used if CI Visibility is enabled and the tracer writes to CI Test Cycle intake

Motivation

The trace agent and intake made certain assumptions that a trace segment would be sent as an atom. The main thing this impacts is likely the service layer metrics. While PendingTrace does not always guarantee that a trace will be written atomically, it makes a best effort to maintain atomicity whenever possible.

The above concerns are not relevant for CI Visibility if tests data is written to CI Test Cycle intake, and not to the APM intake (which for CI VIs is a legacy case that is only used with older DD agents). At the same time buffering traces may lead to losing tests data in some edge cases (namely for traces with continuations which finish after the last test has completed). Taking this into account, the logic was reworked to simply stream spans instead of buffering for CI VIsibility.

Jira ticket: SDTEST-460

[pr-commenter[bot]]

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/ci-visibility-stream-spans
git_commit_date	1720015951	1720083834
git_commit_sha	1496a6cfd7	a9cfc09c04
release_version	1.37.0-SNAPSHOT~1496a6cfd7	1.37.0-SNAPSHOT~a9cfc09c04

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1720087586	1720087586
ci_job_id	562264651	562264651
ci_pipeline_id	38357533	38357533
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 48 metrics, 15 unstable metrics.

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.37.0-SNAPSHOT~a9cfc09c04, baseline=1.37.0-SNAPSHOT~1496a6cfd7

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.064 s) : 0, 1063922
Total [baseline] (8.534 s) : 0, 8533623
Agent [candidate] (1.065 s) : 0, 1065114
Total [candidate] (8.54 s) : 0, 8540241
section iast
Agent [baseline] (1.19 s) : 0, 1190394
Total [baseline] (9.003 s) : 0, 9002959
Agent [candidate] (1.174 s) : 0, 1173875
Total [candidate] (9.047 s) : 0, 9047190
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.173 s) : 0, 1172673
Total [baseline] (8.975 s) : 0, 8975066
Agent [candidate] (1.181 s) : 0, 1180581
Total [candidate] (8.99 s) : 0, 8990374
section iast_TELEMETRY_OFF
Agent [baseline] (1.167 s) : 0, 1166546
Total [baseline] (8.981 s) : 0, 8981086
Agent [candidate] (1.177 s) : 0, 1176750
Total [candidate] (9.059 s) : 0, 9058587

• baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.064 s	-
Agent	iast	1.19 s	126.472 ms (11.9%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.173 s	108.751 ms (10.2%)
Agent	iast_TELEMETRY_OFF	1.167 s	102.624 ms (9.6%)
Total	tracing	8.534 s	-
Total	iast	9.003 s	469.336 ms (5.5%)
Total	iast_HARDCODED_SECRET_DISABLED	8.975 s	441.443 ms (5.2%)
Total	iast_TELEMETRY_OFF	8.981 s	447.462 ms (5.2%)

• candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.065 s	-
Agent	iast	1.174 s	108.761 ms (10.2%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.181 s	115.467 ms (10.8%)
Agent	iast_TELEMETRY_OFF	1.177 s	111.636 ms (10.5%)
Total	tracing	8.54 s	-
Total	iast	9.047 s	506.949 ms (5.9%)
Total	iast_HARDCODED_SECRET_DISABLED	8.99 s	450.133 ms (5.3%)
Total	iast_TELEMETRY_OFF	9.059 s	518.346 ms (6.1%)

gantt
    title insecure-bank - break down per module: candidate=1.37.0-SNAPSHOT~a9cfc09c04, baseline=1.37.0-SNAPSHOT~1496a6cfd7

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (666.08 ms) : 0, 666080
BytebuddyAgent [candidate] (666.153 ms) : 0, 666153
GlobalTracer [baseline] (304.614 ms) : 0, 304614
GlobalTracer [candidate] (305.465 ms) : 0, 305465
AppSec [baseline] (50.361 ms) : 0, 50361
AppSec [candidate] (50.389 ms) : 0, 50389
Remote Config [baseline] (720.104 µs) : 0, 720
Remote Config [candidate] (795.28 µs) : 0, 795
Telemetry [baseline] (7.57 ms) : 0, 7570
Telemetry [candidate] (7.743 ms) : 0, 7743
section iast
BytebuddyAgent [baseline] (795.062 ms) : 0, 795062
BytebuddyAgent [candidate] (780.698 ms) : 0, 780698
GlobalTracer [baseline] (298.949 ms) : 0, 298949
GlobalTracer [candidate] (295.86 ms) : 0, 295860
AppSec [baseline] (47.773 ms) : 0, 47773
AppSec [candidate] (47.526 ms) : 0, 47526
IAST [baseline] (27.318 ms) : 0, 27318
IAST [candidate] (28.033 ms) : 0, 28033
Remote Config [baseline] (577.527 µs) : 0, 578
Remote Config [candidate] (594.21 µs) : 0, 594
Telemetry [baseline] (7.064 ms) : 0, 7064
Telemetry [candidate] (7.772 ms) : 0, 7772
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (780.719 ms) : 0, 780719
BytebuddyAgent [candidate] (785.323 ms) : 0, 785323
GlobalTracer [baseline] (294.924 ms) : 0, 294924
GlobalTracer [candidate] (297.474 ms) : 0, 297474
AppSec [baseline] (47.563 ms) : 0, 47563
AppSec [candidate] (47.613 ms) : 0, 47613
IAST [baseline] (28.368 ms) : 0, 28368
IAST [candidate] (28.155 ms) : 0, 28155
Remote Config [baseline] (604.483 µs) : 0, 604
Remote Config [candidate] (604.183 µs) : 0, 604
Telemetry [baseline] (7.003 ms) : 0, 7003
Telemetry [candidate] (7.852 ms) : 0, 7852
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (777.93 ms) : 0, 777930
BytebuddyAgent [candidate] (783.047 ms) : 0, 783047
GlobalTracer [baseline] (293.842 ms) : 0, 293842
GlobalTracer [candidate] (297.37 ms) : 0, 297370
AppSec [baseline] (46.746 ms) : 0, 46746
AppSec [candidate] (47.39 ms) : 0, 47390
IAST [baseline] (27.08 ms) : 0, 27080
IAST [candidate] (27.821 ms) : 0, 27821
Remote Config [baseline] (617.648 µs) : 0, 618
Remote Config [candidate] (655.348 µs) : 0, 655
Telemetry [baseline] (6.848 ms) : 0, 6848
Telemetry [candidate] (6.948 ms) : 0, 6948

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.37.0-SNAPSHOT~a9cfc09c04, baseline=1.37.0-SNAPSHOT~1496a6cfd7

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.066 s) : 0, 1065818
Total [baseline] (10.411 s) : 0, 10410870
Agent [candidate] (1.064 s) : 0, 1064166
Total [candidate] (10.337 s) : 0, 10336972
section appsec
Agent [baseline] (1.181 s) : 0, 1181251
Total [baseline] (10.558 s) : 0, 10558435
Agent [candidate] (1.183 s) : 0, 1182637
Total [candidate] (10.464 s) : 0, 10463672
section iast
Agent [baseline] (1.193 s) : 0, 1192888
Total [baseline] (10.75 s) : 0, 10750413
Agent [candidate] (1.184 s) : 0, 1183754
Total [candidate] (10.772 s) : 0, 10772471
section profiling
Agent [baseline] (1.265 s) : 0, 1264843
Total [baseline] (10.553 s) : 0, 10552742
Agent [candidate] (1.265 s) : 0, 1265114
Total [candidate] (10.561 s) : 0, 10560934

• baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.066 s	-
Agent	appsec	1.181 s	115.433 ms (10.8%)
Agent	iast	1.193 s	127.07 ms (11.9%)
Agent	profiling	1.265 s	199.025 ms (18.7%)
Total	tracing	10.411 s	-
Total	appsec	10.558 s	147.565 ms (1.4%)
Total	iast	10.75 s	339.542 ms (3.3%)
Total	profiling	10.553 s	141.872 ms (1.4%)

• candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.064 s	-
Agent	appsec	1.183 s	118.471 ms (11.1%)
Agent	iast	1.184 s	119.587 ms (11.2%)
Agent	profiling	1.265 s	200.948 ms (18.9%)
Total	tracing	10.337 s	-
Total	appsec	10.464 s	126.699 ms (1.2%)
Total	iast	10.772 s	435.499 ms (4.2%)
Total	profiling	10.561 s	223.962 ms (2.2%)

gantt
    title petclinic - break down per module: candidate=1.37.0-SNAPSHOT~a9cfc09c04, baseline=1.37.0-SNAPSHOT~1496a6cfd7

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (667.154 ms) : 0, 667154
BytebuddyAgent [candidate] (665.593 ms) : 0, 665593
GlobalTracer [baseline] (305.325 ms) : 0, 305325
GlobalTracer [candidate] (305.566 ms) : 0, 305566
AppSec [baseline] (50.239 ms) : 0, 50239
AppSec [candidate] (50.105 ms) : 0, 50105
Remote Config [baseline] (751.818 µs) : 0, 752
Remote Config [candidate] (697.667 µs) : 0, 698
Telemetry [baseline] (7.578 ms) : 0, 7578
Telemetry [candidate] (7.586 ms) : 0, 7586
section appsec
BytebuddyAgent [baseline] (675.724 ms) : 0, 675724
BytebuddyAgent [candidate] (675.717 ms) : 0, 675717
GlobalTracer [baseline] (297.873 ms) : 0, 297873
GlobalTracer [candidate] (298.681 ms) : 0, 298681
AppSec [baseline] (153.635 ms) : 0, 153635
AppSec [candidate] (153.863 ms) : 0, 153863
IAST [baseline] (21.529 ms) : 0, 21529
IAST [candidate] (21.564 ms) : 0, 21564
Remote Config [baseline] (632.286 µs) : 0, 632
Remote Config [candidate] (630.882 µs) : 0, 631
Telemetry [baseline] (8.05 ms) : 0, 8050
Telemetry [candidate] (8.341 ms) : 0, 8341
section iast
BytebuddyAgent [baseline] (796.076 ms) : 0, 796076
BytebuddyAgent [candidate] (787.68 ms) : 0, 787680
GlobalTracer [baseline] (299.172 ms) : 0, 299172
GlobalTracer [candidate] (297.752 ms) : 0, 297752
AppSec [baseline] (47.824 ms) : 0, 47824
AppSec [candidate] (47.8 ms) : 0, 47800
IAST [baseline] (28.463 ms) : 0, 28463
IAST [candidate] (29.263 ms) : 0, 29263
Remote Config [baseline] (586.385 µs) : 0, 586
Remote Config [candidate] (584.417 µs) : 0, 584
Telemetry [baseline] (7.11 ms) : 0, 7110
Telemetry [candidate] (7.102 ms) : 0, 7102
section profiling
BytebuddyAgent [baseline] (662.955 ms) : 0, 662955
BytebuddyAgent [candidate] (663.129 ms) : 0, 663129
GlobalTracer [baseline] (388.755 ms) : 0, 388755
GlobalTracer [candidate] (389.114 ms) : 0, 389114
AppSec [baseline] (51.659 ms) : 0, 51659
AppSec [candidate] (51.778 ms) : 0, 51778
Remote Config [baseline] (667.642 µs) : 0, 668
Remote Config [candidate] (655.034 µs) : 0, 655
Telemetry [baseline] (7.346 ms) : 0, 7346
Telemetry [candidate] (7.455 ms) : 0, 7455
ProfilingAgent [baseline] (96.252 ms) : 0, 96252
ProfilingAgent [candidate] (95.835 ms) : 0, 95835
Profiling [baseline] (96.277 ms) : 0, 96277
Profiling [candidate] (95.859 ms) : 0, 95859

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2024-07-04T09:33:46	2024-07-04T09:40:34
git_branch	master	nikita-tkachenko/ci-visibility-stream-spans
git_commit_date	1720015951	1720083834
git_commit_sha	1496a6cfd7	a9cfc09c04
release_version	1.37.0-SNAPSHOT~1496a6cfd7	1.37.0-SNAPSHOT~a9cfc09c04
start_time	2024-07-04T09:33:33	2024-07-04T09:40:21

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1720086378	1720086378
ci_job_id	562264652	562264652
ci_pipeline_id	38357533	38357533
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	iast	iast

Summary

Found 1 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 17 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:petclinic:profiling	better [-90.382µs; -35.763µs] or [-5.990%; -2.370%]	unstable [-511.123op/s; +641.437op/s] or [-16.611%; +20.847%]	1.446ms	3142.080op/s	1.509ms	3076.923op/s

Request duration reports for petclinic

gantt
    title petclinic - request duration [CI 0.99] : candidate=1.37.0-SNAPSHOT~a9cfc09c04, baseline=1.37.0-SNAPSHOT~1496a6cfd7
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.352 ms) : 1332, 1371
.   : milestone, 1352,
appsec (1.723 ms) : 1698, 1748
.   : milestone, 1723,
appsec_no_iast (1.724 ms) : 1698, 1749
.   : milestone, 1724,
iast (1.477 ms) : 1454, 1500
.   : milestone, 1477,
profiling (1.509 ms) : 1483, 1535
.   : milestone, 1509,
tracing (1.464 ms) : 1440, 1489
.   : milestone, 1464,
section candidate
no_agent (1.349 ms) : 1330, 1369
.   : milestone, 1349,
appsec (1.702 ms) : 1677, 1726
.   : milestone, 1702,
appsec_no_iast (1.702 ms) : 1678, 1727
.   : milestone, 1702,
iast (1.472 ms) : 1448, 1495
.   : milestone, 1472,
profiling (1.446 ms) : 1421, 1471
.   : milestone, 1446,
tracing (1.458 ms) : 1434, 1483
.   : milestone, 1458,

• baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.352 ms [1.332 ms, 1.371 ms]	-
appsec	1.723 ms [1.698 ms, 1.748 ms]	371.433 µs (27.5%)
appsec_no_iast	1.724 ms [1.698 ms, 1.749 ms]	371.85 µs (27.5%)
iast	1.477 ms [1.454 ms, 1.5 ms]	124.899 µs (9.2%)
profiling	1.509 ms [1.483 ms, 1.535 ms]	157.296 µs (11.6%)
tracing	1.464 ms [1.44 ms, 1.489 ms]	112.772 µs (8.3%)

• candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	1.349 ms [1.33 ms, 1.369 ms]	-
appsec	1.702 ms [1.677 ms, 1.726 ms]	352.236 µs (26.1%)
appsec_no_iast	1.702 ms [1.678 ms, 1.727 ms]	353.019 µs (26.2%)
iast	1.472 ms [1.448 ms, 1.495 ms]	122.273 µs (9.1%)
profiling	1.446 ms [1.421 ms, 1.471 ms]	96.533 µs (7.2%)
tracing	1.458 ms [1.434 ms, 1.483 ms]	109.021 µs (8.1%)

Request duration reports for insecure-bank

gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.37.0-SNAPSHOT~a9cfc09c04, baseline=1.37.0-SNAPSHOT~1496a6cfd7
    dateFormat X
    axisFormat %s
section baseline
no_agent (372.545 µs) : 352, 393
.   : milestone, 373,
iast (475.45 µs) : 455, 496
.   : milestone, 475,
iast_FULL (550.655 µs) : 529, 572
.   : milestone, 551,
iast_GLOBAL (502.179 µs) : 481, 524
.   : milestone, 502,
iast_HARDCODED_SECRET_DISABLED (470.863 µs) : 450, 492
.   : milestone, 471,
iast_INACTIVE (447.619 µs) : 427, 469
.   : milestone, 448,
iast_TELEMETRY_OFF (462.659 µs) : 441, 484
.   : milestone, 463,
tracing (429.28 µs) : 409, 449
.   : milestone, 429,
section candidate
no_agent (367.899 µs) : 348, 388
.   : milestone, 368,
iast (475.319 µs) : 454, 496
.   : milestone, 475,
iast_FULL (549.374 µs) : 528, 570
.   : milestone, 549,
iast_GLOBAL (498.769 µs) : 478, 520
.   : milestone, 499,
iast_HARDCODED_SECRET_DISABLED (482.62 µs) : 461, 505
.   : milestone, 483,
iast_INACTIVE (450.149 µs) : 429, 472
.   : milestone, 450,
iast_TELEMETRY_OFF (467.117 µs) : 446, 488
.   : milestone, 467,
tracing (438.082 µs) : 418, 458
.   : milestone, 438,

• baseline results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	372.545 µs [352.443 µs, 392.647 µs]	-
iast	475.45 µs [454.545 µs, 496.355 µs]	102.905 µs (27.6%)
iast_FULL	550.655 µs [529.374 µs, 571.937 µs]	178.111 µs (47.8%)
iast_GLOBAL	502.179 µs [480.531 µs, 523.828 µs]	129.635 µs (34.8%)
iast_HARDCODED_SECRET_DISABLED	470.863 µs [450.104 µs, 491.621 µs]	98.318 µs (26.4%)
iast_INACTIVE	447.619 µs [426.592 µs, 468.645 µs]	75.074 µs (20.2%)
iast_TELEMETRY_OFF	462.659 µs [441.448 µs, 483.87 µs]	90.114 µs (24.2%)
tracing	429.28 µs [409.069 µs, 449.491 µs]	56.736 µs (15.2%)

• candidate results

Variant	Request duration [CI 0.99]	Δ no_agent
no_agent	367.899 µs [348.16 µs, 387.638 µs]	-
iast	475.319 µs [454.211 µs, 496.427 µs]	107.42 µs (29.2%)
iast_FULL	549.374 µs [528.385 µs, 570.363 µs]	181.475 µs (49.3%)
iast_GLOBAL	498.769 µs [477.781 µs, 519.757 µs]	130.87 µs (35.6%)
iast_HARDCODED_SECRET_DISABLED	482.62 µs [460.528 µs, 504.712 µs]	114.721 µs (31.2%)
iast_INACTIVE	450.149 µs [428.743 µs, 471.555 µs]	82.25 µs (22.4%)
iast_TELEMETRY_OFF	467.117 µs [445.897 µs, 488.338 µs]	99.218 µs (27.0%)
tracing	438.082 µs [417.776 µs, 458.389 µs]	70.184 µs (19.1%)

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	nikita-tkachenko/ci-visibility-stream-spans
git_commit_date	1720015951	1720083834
git_commit_sha	1496a6cfd7	a9cfc09c04
release_version	1.37.0-SNAPSHOT~1496a6cfd7	1.37.0-SNAPSHOT~a9cfc09c04

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1720087105	1720087105
ci_job_id	562264653	562264653
ci_pipeline_id	38357533	38357533
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.37.0-SNAPSHOT~a9cfc09c04, baseline=1.37.0-SNAPSHOT~1496a6cfd7
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.455 ms) : 1444, 1466
.   : milestone, 1455,
appsec (2.24 ms) : 2204, 2277
.   : milestone, 2240,
iast (1.995 ms) : 1951, 2039
.   : milestone, 1995,
iast_GLOBAL (2.026 ms) : 1981, 2070
.   : milestone, 2026,
profiling (1.869 ms) : 1834, 1904
.   : milestone, 1869,
tracing (1.852 ms) : 1819, 1886
.   : milestone, 1852,
section candidate
no_agent (1.469 ms) : 1457, 1481
.   : milestone, 1469,
appsec (2.21 ms) : 2176, 2244
.   : milestone, 2210,
iast (1.969 ms) : 1928, 2010
.   : milestone, 1969,
iast_GLOBAL (2.013 ms) : 1971, 2055
.   : milestone, 2013,
profiling (1.85 ms) : 1817, 1884
.   : milestone, 1850,
tracing (1.837 ms) : 1804, 1869
.   : milestone, 1837,

• baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.455 ms [1.444 ms, 1.466 ms]	-
appsec	2.24 ms [2.204 ms, 2.277 ms]	785.173 µs (54.0%)
iast	1.995 ms [1.951 ms, 2.039 ms]	539.968 µs (37.1%)
iast_GLOBAL	2.026 ms [1.981 ms, 2.07 ms]	570.292 µs (39.2%)
profiling	1.869 ms [1.834 ms, 1.904 ms]	413.979 µs (28.4%)
tracing	1.852 ms [1.819 ms, 1.886 ms]	397.261 µs (27.3%)

• candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.469 ms [1.457 ms, 1.481 ms]	-
appsec	2.21 ms [2.176 ms, 2.244 ms]	741.224 µs (50.5%)
iast	1.969 ms [1.928 ms, 2.01 ms]	500.133 µs (34.0%)
iast_GLOBAL	2.013 ms [1.971 ms, 2.055 ms]	544.022 µs (37.0%)
profiling	1.85 ms [1.817 ms, 1.884 ms]	381.2 µs (25.9%)
tracing	1.837 ms [1.804 ms, 1.869 ms]	367.5 µs (25.0%)

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.37.0-SNAPSHOT~a9cfc09c04, baseline=1.37.0-SNAPSHOT~1496a6cfd7
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.977 s) : 14977000, 14977000
.   : milestone, 14977000,
appsec (14.918 s) : 14918000, 14918000
.   : milestone, 14918000,
iast (18.72 s) : 18720000, 18720000
.   : milestone, 18720000,
iast_GLOBAL (18.1 s) : 18100000, 18100000
.   : milestone, 18100000,
profiling (15.062 s) : 15062000, 15062000
.   : milestone, 15062000,
tracing (15.268 s) : 15268000, 15268000
.   : milestone, 15268000,
section candidate
no_agent (15.403 s) : 15403000, 15403000
.   : milestone, 15403000,
appsec (14.931 s) : 14931000, 14931000
.   : milestone, 14931000,
iast (18.768 s) : 18768000, 18768000
.   : milestone, 18768000,
iast_GLOBAL (17.762 s) : 17762000, 17762000
.   : milestone, 17762000,
profiling (15.057 s) : 15057000, 15057000
.   : milestone, 15057000,
tracing (14.763 s) : 14763000, 14763000
.   : milestone, 14763000,

• baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.977 s [14.977 s, 14.977 s]	-
appsec	14.918 s [14.918 s, 14.918 s]	-59.0 ms (-0.4%)
iast	18.72 s [18.72 s, 18.72 s]	3.743 s (25.0%)
iast_GLOBAL	18.1 s [18.1 s, 18.1 s]	3.123 s (20.9%)
profiling	15.062 s [15.062 s, 15.062 s]	85.0 ms (0.6%)
tracing	15.268 s [15.268 s, 15.268 s]	291.0 ms (1.9%)

• candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.403 s [15.403 s, 15.403 s]	-
appsec	14.931 s [14.931 s, 14.931 s]	-472.0 ms (-3.1%)
iast	18.768 s [18.768 s, 18.768 s]	3.365 s (21.8%)
iast_GLOBAL	17.762 s [17.762 s, 17.762 s]	2.359 s (15.3%)
profiling	15.057 s [15.057 s, 15.057 s]	-346.0 ms (-2.2%)
tracing	14.763 s [14.763 s, 14.763 s]	-640.0 ms (-4.2%)

[mcculls]

Reminder to get approval from the platform team before merging