Llmobs java sdk integration

Open gary-huang opened this issue 5 months ago • 1 comments

trafficstars

What Does This Do

so we can get a nicer s3 link

Motivation

Additional Notes

Contributor Checklist

Format the title according the contribution guidelines
Assign the type: and (comp: or inst:) labels in addition to any usefull labels
Don't use close, fix or any linking keywords when referencing an issue.
Use solves instead, and assign the PR milestone to the issue
Update the CODEOWNERS file on source file addition, move, or deletion
Update the public documentation in case of new configuration flag or behavior

Jira ticket: [PROJ-IDENT]

Jun 11 '25 17:06 gary-huang

Benchmarks

Startup

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	llmobs-java-sdk-integration
git_commit_date	1749638913	1749600946
git_commit_sha	089cf8fd47	8659686578
release_version	1.50.0-SNAPSHOT~089cf8fd47	1.50.0-SNAPSHOT~8659686578

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1749667271	1749667271
ci_job_id	977242415	977242415
ci_pipeline_id	67452334	67452334
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-2ousnmcz-project-304-concurrent-1-9nrbd13k 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-2ousnmcz-project-304-concurrent-1-9nrbd13k 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module	Agent	Agent
parent	None	None
variant	iast	iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 59 metrics, 12 unstable metrics.

Startup time reports for petclinic

gantt
    title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~8659686578, baseline=1.50.0-SNAPSHOT~089cf8fd47

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.024 s) : 0, 1023996
Total [baseline] (11.174 s) : 0, 11173840
Agent [candidate] (1.029 s) : 0, 1028861
Total [candidate] (11.061 s) : 0, 11061183
section appsec
Agent [baseline] (1.165 s) : 0, 1164856
Total [baseline] (11.227 s) : 0, 11226705
Agent [candidate] (1.162 s) : 0, 1162244
Total [candidate] (11.177 s) : 0, 11176827
section iast
Agent [baseline] (1.15 s) : 0, 1149767
Total [baseline] (11.314 s) : 0, 11313947
Agent [candidate] (1.157 s) : 0, 1157420
Total [candidate] (11.364 s) : 0, 11363782
section profiling
Agent [baseline] (1.264 s) : 0, 1263884
Total [baseline] (11.408 s) : 0, 11408030
Agent [candidate] (1.274 s) : 0, 1273931
Total [candidate] (11.452 s) : 0, 11451862

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.024 s	-
Agent	appsec	1.165 s	140.86 ms (13.8%)
Agent	iast	1.15 s	125.772 ms (12.3%)
Agent	profiling	1.264 s	239.889 ms (23.4%)
Total	tracing	11.174 s	-
Total	appsec	11.227 s	52.865 ms (0.5%)
Total	iast	11.314 s	140.107 ms (1.3%)
Total	profiling	11.408 s	234.191 ms (2.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.029 s	-
Agent	appsec	1.162 s	133.383 ms (13.0%)
Agent	iast	1.157 s	128.559 ms (12.5%)
Agent	profiling	1.274 s	245.07 ms (23.8%)
Total	tracing	11.061 s	-
Total	appsec	11.177 s	115.645 ms (1.0%)
Total	iast	11.364 s	302.599 ms (2.7%)
Total	profiling	11.452 s	390.679 ms (3.5%)

gantt
    title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~8659686578, baseline=1.50.0-SNAPSHOT~089cf8fd47

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (684.202 ms) : 0, 684202
BytebuddyAgent [candidate] (688.011 ms) : 0, 688011
GlobalTracer [baseline] (240.808 ms) : 0, 240808
GlobalTracer [candidate] (241.666 ms) : 0, 241666
AppSec [baseline] (57.007 ms) : 0, 57007
AppSec [candidate] (56.453 ms) : 0, 56453
Debugger [baseline] (6.191 ms) : 0, 6191
Debugger [candidate] (6.141 ms) : 0, 6141
Remote Config [baseline] (730.381 µs) : 0, 730
Remote Config [candidate] (734.104 µs) : 0, 734
Telemetry [baseline] (11.387 ms) : 0, 11387
Telemetry [candidate] (12.125 ms) : 0, 12125
section appsec
BytebuddyAgent [baseline] (701.607 ms) : 0, 701607
BytebuddyAgent [candidate] (700.094 ms) : 0, 700094
GlobalTracer [baseline] (238.171 ms) : 0, 238171
GlobalTracer [candidate] (237.716 ms) : 0, 237716
AppSec [baseline] (176.467 ms) : 0, 176467
AppSec [candidate] (176.092 ms) : 0, 176092
Debugger [baseline] (5.987 ms) : 0, 5987
Debugger [candidate] (5.958 ms) : 0, 5958
Remote Config [baseline] (635.006 µs) : 0, 635
Remote Config [candidate] (628.183 µs) : 0, 628
Telemetry [baseline] (7.392 ms) : 0, 7392
Telemetry [candidate] (7.32 ms) : 0, 7320
IAST [baseline] (22.05 ms) : 0, 22050
IAST [candidate] (21.794 ms) : 0, 21794
section iast
BytebuddyAgent [baseline] (801.675 ms) : 0, 801675
BytebuddyAgent [candidate] (806.659 ms) : 0, 806659
GlobalTracer [baseline] (230.516 ms) : 0, 230516
GlobalTracer [candidate] (231.318 ms) : 0, 231318
AppSec [baseline] (51.107 ms) : 0, 51107
AppSec [candidate] (52.697 ms) : 0, 52697
Debugger [baseline] (6.038 ms) : 0, 6038
Debugger [candidate] (6.088 ms) : 0, 6088
Remote Config [baseline] (597.004 µs) : 0, 597
Remote Config [candidate] (611.322 µs) : 0, 611
Telemetry [baseline] (7.926 ms) : 0, 7926
Telemetry [candidate] (8.03 ms) : 0, 8030
IAST [baseline] (28.449 ms) : 0, 28449
IAST [candidate] (28.185 ms) : 0, 28185
section profiling
BytebuddyAgent [baseline] (674.62 ms) : 0, 674620
BytebuddyAgent [candidate] (680.739 ms) : 0, 680739
GlobalTracer [baseline] (359.656 ms) : 0, 359656
GlobalTracer [candidate] (361.343 ms) : 0, 361343
AppSec [baseline] (61.363 ms) : 0, 61363
AppSec [candidate] (61.844 ms) : 0, 61844
Debugger [baseline] (6.069 ms) : 0, 6069
Debugger [candidate] (6.137 ms) : 0, 6137
Remote Config [baseline] (660.489 µs) : 0, 660
Remote Config [candidate] (664.096 µs) : 0, 664
Telemetry [baseline] (8.057 ms) : 0, 8057
Telemetry [candidate] (8.191 ms) : 0, 8191
ProfilingAgent [baseline] (102.702 ms) : 0, 102702
ProfilingAgent [candidate] (103.974 ms) : 0, 103974
Profiling [baseline] (102.726 ms) : 0, 102726
Profiling [candidate] (103.998 ms) : 0, 103998

Startup time reports for insecure-bank

gantt
    title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~8659686578, baseline=1.50.0-SNAPSHOT~089cf8fd47

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.031 s) : 0, 1030770
Total [baseline] (8.543 s) : 0, 8542628
Agent [candidate] (1.023 s) : 0, 1022796
Total [candidate] (8.493 s) : 0, 8492543
section iast
Agent [baseline] (1.15 s) : 0, 1149996
Total [baseline] (9.16 s) : 0, 9160209
Agent [candidate] (1.152 s) : 0, 1151668
Total [candidate] (9.165 s) : 0, 9165072
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.156 s) : 0, 1156342
Total [baseline] (9.142 s) : 0, 9142016
Agent [candidate] (1.154 s) : 0, 1153536
Total [candidate] (9.128 s) : 0, 9128049
section iast_TELEMETRY_OFF
Agent [baseline] (1.15 s) : 0, 1150397
Total [baseline] (9.153 s) : 0, 9153232
Agent [candidate] (1.155 s) : 0, 1155423
Total [candidate] (9.222 s) : 0, 9222070

baseline results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.031 s	-
Agent	iast	1.15 s	119.226 ms (11.6%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.156 s	125.572 ms (12.2%)
Agent	iast_TELEMETRY_OFF	1.15 s	119.626 ms (11.6%)
Total	tracing	8.543 s	-
Total	iast	9.16 s	617.581 ms (7.2%)
Total	iast_HARDCODED_SECRET_DISABLED	9.142 s	599.388 ms (7.0%)
Total	iast_TELEMETRY_OFF	9.153 s	610.603 ms (7.1%)

candidate results

Module	Variant	Duration	Δ tracing
Agent	tracing	1.023 s	-
Agent	iast	1.152 s	128.872 ms (12.6%)
Agent	iast_HARDCODED_SECRET_DISABLED	1.154 s	130.74 ms (12.8%)
Agent	iast_TELEMETRY_OFF	1.155 s	132.627 ms (13.0%)
Total	tracing	8.493 s	-
Total	iast	9.165 s	672.529 ms (7.9%)
Total	iast_HARDCODED_SECRET_DISABLED	9.128 s	635.506 ms (7.5%)
Total	iast_TELEMETRY_OFF	9.222 s	729.526 ms (8.6%)

gantt
    title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~8659686578, baseline=1.50.0-SNAPSHOT~089cf8fd47

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (688.337 ms) : 0, 688337
BytebuddyAgent [candidate] (683.899 ms) : 0, 683899
GlobalTracer [baseline] (241.292 ms) : 0, 241292
GlobalTracer [candidate] (240.262 ms) : 0, 240262
AppSec [baseline] (57.778 ms) : 0, 57778
AppSec [candidate] (57.522 ms) : 0, 57522
Debugger [baseline] (6.168 ms) : 0, 6168
Debugger [candidate] (6.108 ms) : 0, 6108
Remote Config [baseline] (750.336 µs) : 0, 750
Remote Config [candidate] (737.633 µs) : 0, 738
Telemetry [baseline] (12.722 ms) : 0, 12722
Telemetry [candidate] (10.688 ms) : 0, 10688
section iast
BytebuddyAgent [baseline] (802.525 ms) : 0, 802525
BytebuddyAgent [candidate] (803.607 ms) : 0, 803607
GlobalTracer [baseline] (230.488 ms) : 0, 230488
GlobalTracer [candidate] (231.146 ms) : 0, 231146
IAST [baseline] (28.668 ms) : 0, 28668
IAST [candidate] (27.75 ms) : 0, 27750
AppSec [baseline] (50.348 ms) : 0, 50348
AppSec [candidate] (51.19 ms) : 0, 51190
Debugger [baseline] (5.955 ms) : 0, 5955
Debugger [candidate] (5.938 ms) : 0, 5938
Remote Config [baseline] (601.025 µs) : 0, 601
Remote Config [candidate] (602.618 µs) : 0, 603
Telemetry [baseline] (7.904 ms) : 0, 7904
Telemetry [candidate] (7.877 ms) : 0, 7877
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (806.507 ms) : 0, 806507
BytebuddyAgent [candidate] (804.034 ms) : 0, 804034
GlobalTracer [baseline] (231.335 ms) : 0, 231335
GlobalTracer [candidate] (230.687 ms) : 0, 230687
IAST [baseline] (26.438 ms) : 0, 26438
IAST [candidate] (26.637 ms) : 0, 26637
AppSec [baseline] (53.711 ms) : 0, 53711
AppSec [candidate] (53.984 ms) : 0, 53984
Debugger [baseline] (6.028 ms) : 0, 6028
Debugger [candidate] (5.991 ms) : 0, 5991
Remote Config [baseline] (600.208 µs) : 0, 600
Remote Config [candidate] (591.763 µs) : 0, 592
Telemetry [baseline] (8.035 ms) : 0, 8035
Telemetry [candidate] (7.981 ms) : 0, 7981
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (802.399 ms) : 0, 802399
BytebuddyAgent [candidate] (805.236 ms) : 0, 805236
GlobalTracer [baseline] (230.671 ms) : 0, 230671
GlobalTracer [candidate] (232.66 ms) : 0, 232660
IAST [baseline] (28.167 ms) : 0, 28167
IAST [candidate] (28.208 ms) : 0, 28208
AppSec [baseline] (51.082 ms) : 0, 51082
AppSec [candidate] (50.443 ms) : 0, 50443
Debugger [baseline] (5.985 ms) : 0, 5985
Debugger [candidate] (5.993 ms) : 0, 5993
Remote Config [baseline] (604.56 µs) : 0, 605
Remote Config [candidate] (599.23 µs) : 0, 599
Telemetry [baseline] (7.882 ms) : 0, 7882
Telemetry [candidate] (7.912 ms) : 0, 7912

Load

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
end_time	2025-06-11T18:10:47	2025-06-11T18:26:33
git_branch	master	llmobs-java-sdk-integration
git_commit_date	1749638913	1749600946
git_commit_sha	089cf8fd47	8659686578
release_version	1.50.0-SNAPSHOT~089cf8fd47	1.50.0-SNAPSHOT~8659686578
start_time	2025-06-11T18:10:32	2025-06-11T18:26:18

See matching parameters

	Baseline	Candidate
application	insecure-bank	insecure-bank
ci_job_date	1749667121	1749667121
ci_job_id	977242417	977242417
ci_pipeline_id	67452334	67452334
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-66eemyw1-project-304-concurrent-1-gcgw2fgd 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-66eemyw1-project-304-concurrent-1-gcgw2fgd 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
thresholds_or_results	results	results
variant	iast	iast

Summary

Found 3 performance improvements and 5 performance regressions! Performance is the same for 7 metrics, 13 unstable metrics.

scenario	Δ mean http_req_duration	Δ mean throughput	candidate mean http_req_duration	candidate mean throughput	baseline mean http_req_duration	baseline mean throughput
scenario:load:petclinic:code_origins	worse [+589.926µs; +654.878µs] or [+7.188%; +7.980%]	worse [-59.201op/s; -26.313op/s] or [-9.842%; -4.375%]	8.829ms	558.746op/s	8.207ms	601.504op/s
scenario:load:petclinic:iast	better [-935.808µs; -876.750µs] or [-11.621%; -10.887%]	better [+56.379op/s; +92.162op/s] or [+9.162%; +14.976%]	7.147ms	689.655op/s	8.053ms	615.385op/s
scenario:load:petclinic:no_agent	worse [+743.228µs; +808.667µs] or [+8.489%; +9.236%]	worse [-65.606op/s; -22.194op/s] or [-11.645%; -3.939%]	9.531ms	519.481op/s	8.755ms	563.380op/s
scenario:load:petclinic:profiling	better [-8.824ms; -8.765ms] or [-99.530%; -98.865%]	unstable [+5194.003op/s; +20353.783op/s] or [+928.428%; +3638.239%]	0.071ms	13333.333op/s	8.866ms	559.441op/s
scenario:load:petclinic:tracing	worse [+723.267µs; +786.903µs] or [+inf%; +inf%]	unstable [-20454.792op/s; -2133.444op/s] or [-127.842%; -13.334%]	755084.734ns	4705.882op/s	0.000ns	16000.000op/s

Dacapo

Parameters

	Baseline	Candidate
baseline_or_candidate	baseline	candidate
git_branch	master	llmobs-java-sdk-integration
git_commit_date	1749638913	1749600946
git_commit_sha	089cf8fd47	8659686578
release_version	1.50.0-SNAPSHOT~089cf8fd47	1.50.0-SNAPSHOT~8659686578

See matching parameters

	Baseline	Candidate
application	biojava	biojava
ci_job_date	1749666792	1749666792
ci_job_id	977242419	977242419
ci_pipeline_id	67452334	67452334
cpu_model	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz	Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version	Linux runner-2ousnmcz-project-304-concurrent-2-xyb6f67w 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux	Linux runner-2ousnmcz-project-304-concurrent-2-xyb6f67w 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant	appsec	appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for biojava

gantt
    title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~8659686578, baseline=1.50.0-SNAPSHOT~089cf8fd47
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.969 s) : 14969000, 14969000
.   : milestone, 14969000,
appsec (14.627 s) : 14627000, 14627000
.   : milestone, 14627000,
iast (18.749 s) : 18749000, 18749000
.   : milestone, 18749000,
iast_GLOBAL (17.793 s) : 17793000, 17793000
.   : milestone, 17793000,
profiling (15.763 s) : 15763000, 15763000
.   : milestone, 15763000,
tracing (14.896 s) : 14896000, 14896000
.   : milestone, 14896000,
section candidate
no_agent (15.379 s) : 15379000, 15379000
.   : milestone, 15379000,
appsec (14.834 s) : 14834000, 14834000
.   : milestone, 14834000,
iast (18.525 s) : 18525000, 18525000
.   : milestone, 18525000,
iast_GLOBAL (18.297 s) : 18297000, 18297000
.   : milestone, 18297000,
profiling (15.617 s) : 15617000, 15617000
.   : milestone, 15617000,
tracing (14.965 s) : 14965000, 14965000
.   : milestone, 14965000,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	14.969 s [14.969 s, 14.969 s]	-
appsec	14.627 s [14.627 s, 14.627 s]	-342.0 ms (-2.3%)
iast	18.749 s [18.749 s, 18.749 s]	3.78 s (25.3%)
iast_GLOBAL	17.793 s [17.793 s, 17.793 s]	2.824 s (18.9%)
profiling	15.763 s [15.763 s, 15.763 s]	794.0 ms (5.3%)
tracing	14.896 s [14.896 s, 14.896 s]	-73.0 ms (-0.5%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	15.379 s [15.379 s, 15.379 s]	-
appsec	14.834 s [14.834 s, 14.834 s]	-545.0 ms (-3.5%)
iast	18.525 s [18.525 s, 18.525 s]	3.146 s (20.5%)
iast_GLOBAL	18.297 s [18.297 s, 18.297 s]	2.918 s (19.0%)
profiling	15.617 s [15.617 s, 15.617 s]	238.0 ms (1.5%)
tracing	14.965 s [14.965 s, 14.965 s]	-414.0 ms (-2.7%)

Execution time for tomcat

gantt
    title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~8659686578, baseline=1.50.0-SNAPSHOT~089cf8fd47
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.47 ms) : 1459, 1482
.   : milestone, 1470,
appsec (2.391 ms) : 2343, 2440
.   : milestone, 2391,
iast (2.175 ms) : 2114, 2236
.   : milestone, 2175,
iast_GLOBAL (2.222 ms) : 2160, 2283
.   : milestone, 2222,
profiling (2.032 ms) : 1982, 2082
.   : milestone, 2032,
tracing (1.992 ms) : 1945, 2040
.   : milestone, 1992,
section candidate
no_agent (1.47 ms) : 1459, 1481
.   : milestone, 1470,
appsec (2.384 ms) : 2336, 2433
.   : milestone, 2384,
iast (2.174 ms) : 2113, 2235
.   : milestone, 2174,
iast_GLOBAL (2.225 ms) : 2163, 2286
.   : milestone, 2225,
profiling (2.026 ms) : 1977, 2075
.   : milestone, 2026,
tracing (2.001 ms) : 1954, 2049
.   : milestone, 2001,

baseline results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.47 ms [1.459 ms, 1.482 ms]	-
appsec	2.391 ms [2.343 ms, 2.44 ms]	921.03 µs (62.6%)
iast	2.175 ms [2.114 ms, 2.236 ms]	705.1 µs (48.0%)
iast_GLOBAL	2.222 ms [2.16 ms, 2.283 ms]	751.35 µs (51.1%)
profiling	2.032 ms [1.982 ms, 2.082 ms]	561.694 µs (38.2%)
tracing	1.992 ms [1.945 ms, 2.04 ms]	522.099 µs (35.5%)

candidate results

Variant	Execution Time [CI 0.99]	Δ no_agent
no_agent	1.47 ms [1.459 ms, 1.481 ms]	-
appsec	2.384 ms [2.336 ms, 2.433 ms]	914.492 µs (62.2%)
iast	2.174 ms [2.113 ms, 2.235 ms]	704.309 µs (47.9%)
iast_GLOBAL	2.225 ms [2.163 ms, 2.286 ms]	754.552 µs (51.3%)
profiling	2.026 ms [1.977 ms, 2.075 ms]	556.041 µs (37.8%)
tracing	2.001 ms [1.954 ms, 2.049 ms]	531.404 µs (36.2%)

Jun 11 '25 18:06 pr-commenter[bot]

dd-trace-java dd-trace-java copied to clipboard

Llmobs java sdk integration

What Does This Do

Motivation

Additional Notes

Contributor Checklist

Benchmarks

Startup

Parameters

Summary

Load

Parameters

Summary

Dacapo

Parameters

Summary

dd-trace-java
dd-trace-java copied to clipboard