dd-trace-dotnet icon indicating copy to clipboard operation
dd-trace-dotnet copied to clipboard
verified publisher icon DataDog

[ASM] Standalone Billing

Open e-n-0 opened this issue 1 year ago • 6 comments

Summary of changes

Implementing ASM Standalone Billing without the propagation.

Reason for change

Allow ASM Products to run without APM.

Implementation details

Following the RFC.

  • New env var: DD_EXPERIMENTAL_APPSEC_STANDALONE_ENABLED, disabled by default
  • Add the tag: _dd.apm.enabled:0
  • Force enable the header Datadog-Client-Computed-Stats: yes to be set, but also force the deactivation of StatsComputationEnabled
  • Change the sampling rule to discard APM traces.
    • Only allow 1 trace per minute to keep the service alive in the backend.
    • Trace with IAST event are marked to keep them

Test coverage

  • Unit testing testing:
    • If the env var is correctly applied
    • If the StatsComputationEnabled is correctly disabled if ASM Standalone is enabled
  • Integration test to check if the tag: _dd.apm.enabled:0 is correctly set.

Other details

Related PR to update the config_norm_rules for telemetry.

e-n-0 avatar May 15 '24 15:05 e-n-0

Datadog Report

Branch report: flavien/asm/standalone-billing Commit report: c12982a Test service: dd-trace-dotnet

:white_check_mark: 0 Failed, 437952 Passed, 2697 Skipped, 25h 8m 52.55s Total Time

datadog-ddstaging[bot] avatar May 15 '24 16:05 datadog-ddstaging[bot]

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5565) - mean (74ms)  : 64, 83
     .   : milestone, 74,
    master - mean (74ms)  : 65, 83
     .   : milestone, 74,

    section CallTarget+Inlining+NGEN
    This PR (5565) - mean (1,037ms)  : 1022, 1052
     .   : milestone, 1037,
    master - mean (1,040ms)  : 1019, 1062
     .   : milestone, 1040,


gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5565) - mean (110ms)  : 107, 114
     .   : milestone, 110,
    master - mean (109ms)  : 107, 111
     .   : milestone, 109,

    section CallTarget+Inlining+NGEN
    This PR (5565) - mean (728ms)  : 705, 751
     .   : milestone, 728,
    master - mean (729ms)  : 707, 752
     .   : milestone, 729,


gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5565) - mean (94ms)  : 90, 97
     .   : milestone, 94,
    master - mean (93ms)  : 90, 96
     .   : milestone, 93,

    section CallTarget+Inlining+NGEN
    This PR (5565) - mean (684ms)  : 662, 707
     .   : milestone, 684,
    master - mean (687ms)  : 661, 713
     .   : milestone, 687,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5565) - mean (192ms)  : 189, 194
     .   : milestone, 192,
    master - mean (192ms)  : 189, 196
     .   : milestone, 192,

    section CallTarget+Inlining+NGEN
    This PR (5565) - mean (1,162ms)  : 1136, 1188
     .   : milestone, 1162,
    master - mean (1,164ms)  : 1139, 1188
     .   : milestone, 1164,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5565) - mean (277ms)  : 273, 281
     .   : milestone, 277,
    master - mean (280ms)  : 273, 287
     .   : milestone, 280,

    section CallTarget+Inlining+NGEN
    This PR (5565) - mean (949ms)  : 931, 967
     .   : milestone, 949,
    master - mean (954ms)  : 935, 972
     .   : milestone, 954,


gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5565) - mean (264ms)  : 259, 268
     .   : milestone, 264,
    master - mean (265ms)  : 261, 269
     .   : milestone, 265,

    section CallTarget+Inlining+NGEN
    This PR (5565) - mean (925ms)  : 907, 943
     .   : milestone, 925,
    master - mean (930ms)  : 908, 952
     .   : milestone, 930,

andrewlock avatar May 15 '24 16:05 andrewlock

Benchmarks Report for tracer :snail:

Benchmarks for #5565 compared to master:

  • 2 benchmarks are faster, with geometric mean 1.137
  • 1 benchmarks are slower, with geometric mean 1.138
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 7.83μs 44.5ns 302ns 0.0159 0.00797 0 5.43 KB
master StartStopWithChild netcoreapp3.1 9.8μs 51.8ns 254ns 0.0193 0.00965 0 5.62 KB
master StartStopWithChild net472 16.1μs 43.2ns 162ns 1 0.291 0.0807 6.07 KB
#5565 StartStopWithChild net6.0 7.78μs 41.3ns 219ns 0.0196 0.00784 0 5.42 KB
#5565 StartStopWithChild netcoreapp3.1 10μs 55.2ns 358ns 0.0186 0.00932 0 5.61 KB
#5565 StartStopWithChild net472 16.3μs 42.6ns 165ns 1.01 0.298 0.0995 6.07 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 487μs 352ns 1.36μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 648μs 483ns 1.81μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 837μs 441ns 1.71μs 0.414 0 0 3.3 KB
#5565 WriteAndFlushEnrichedTraces net6.0 478μs 288ns 1.12μs 0 0 0 2.7 KB
#5565 WriteAndFlushEnrichedTraces netcoreapp3.1 641μs 279ns 1.08μs 0 0 0 2.7 KB
#5565 WriteAndFlushEnrichedTraces net472 843μs 450ns 1.74μs 0.419 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 172μs 186ns 720ns 0.258 0 0 18.45 KB
master SendRequest netcoreapp3.1 193μs 295ns 1.14μs 0.193 0 0 20.61 KB
master SendRequest net472 0.000369ns 0.000144ns 0.00054ns 0 0 0 0 b
#5565 SendRequest net6.0 171μs 237ns 917ns 0.171 0 0 18.45 KB
#5565 SendRequest netcoreapp3.1 196μs 346ns 1.3μs 0.195 0 0 20.61 KB
#5565 SendRequest net472 6.96E‑05ns 6.96E‑05ns 0.000251ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 549μs 729ns 2.82μs 0.536 0 0 41.63 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 647μs 886ns 3.43μs 0.329 0 0 41.94 KB
master WriteAndFlushEnrichedTraces net472 847μs 3.06μs 11.9μs 8.08 2.55 0.425 53.2 KB
#5565 WriteAndFlushEnrichedTraces net6.0 540μs 172ns 645ns 0.532 0 0 41.43 KB
#5565 WriteAndFlushEnrichedTraces netcoreapp3.1 662μs 792ns 3.07μs 0.331 0 0 41.77 KB
#5565 WriteAndFlushEnrichedTraces net472 822μs 4.14μs 17.6μs 8.33 2.5 0.417 53.22 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.31μs 0.376ns 1.41ns 0.0145 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 1.7μs 1.19ns 4.62ns 0.0137 0 0 1.02 KB
master ExecuteNonQuery net472 1.93μs 3.65ns 14.2ns 0.156 0 0 987 B
#5565 ExecuteNonQuery net6.0 1.23μs 0.589ns 2.28ns 0.0143 0 0 1.02 KB
#5565 ExecuteNonQuery netcoreapp3.1 1.75μs 0.957ns 3.58ns 0.0142 0 0 1.02 KB
#5565 ExecuteNonQuery net472 1.92μs 0.605ns 2.34ns 0.156 0 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Faster :tada: Same allocations :heavy_check_mark:

Faster :tada: in #5565

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch‑net6.0 1.156 1,331.87 1,152.16

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.33μs 0.53ns 1.98ns 0.0135 0 0 976 B
master CallElasticsearch netcoreapp3.1 1.54μs 0.71ns 2.75ns 0.0131 0 0 976 B
master CallElasticsearch net472 2.46μs 1.9ns 7.37ns 0.157 0.00123 0 995 B
master CallElasticsearchAsync net6.0 1.2μs 0.487ns 1.82ns 0.0131 0 0 952 B
master CallElasticsearchAsync netcoreapp3.1 1.62μs 3.07ns 11.1ns 0.0137 0 0 1.02 KB
master CallElasticsearchAsync net472 2.69μs 1.55ns 5.79ns 0.167 0.00134 0 1.05 KB
#5565 CallElasticsearch net6.0 1.15μs 0.54ns 2.09ns 0.0138 0 0 976 B
#5565 CallElasticsearch netcoreapp3.1 1.48μs 0.849ns 3.18ns 0.0133 0 0 976 B
#5565 CallElasticsearch net472 2.45μs 2.01ns 7.52ns 0.158 0 0 995 B
#5565 CallElasticsearchAsync net6.0 1.13μs 0.216ns 0.777ns 0.0136 0 0 952 B
#5565 CallElasticsearchAsync netcoreapp3.1 1.62μs 2.12ns 7.92ns 0.0135 0 0 1.02 KB
#5565 CallElasticsearchAsync net472 2.71μs 2.32ns 8.68ns 0.166 0.00135 0 1.05 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.17μs 0.638ns 2.39ns 0.0134 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.55μs 0.981ns 3.8ns 0.0129 0 0 952 B
master ExecuteAsync net472 1.84μs 1.32ns 5.11ns 0.145 0 0 915 B
#5565 ExecuteAsync net6.0 1.21μs 0.684ns 2.65ns 0.0134 0 0 952 B
#5565 ExecuteAsync netcoreapp3.1 1.59μs 1.06ns 4.12ns 0.0128 0 0 952 B
#5565 ExecuteAsync net472 1.84μs 1.19ns 4.6ns 0.145 0.000918 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.17μs 1.63ns 6.08ns 0.03 0 0 2.22 KB
master SendAsync netcoreapp3.1 5.28μs 3.23ns 11.2ns 0.037 0 0 2.76 KB
master SendAsync net472 7.8μs 2.7ns 10.4ns 0.497 0 0 3.15 KB
#5565 SendAsync net6.0 4.12μs 1.55ns 5.79ns 0.031 0 0 2.22 KB
#5565 SendAsync netcoreapp3.1 5.03μs 2.5ns 9.68ns 0.0353 0 0 2.76 KB
#5565 SendAsync net472 7.7μs 2.7ns 10.5ns 0.499 0 0 3.15 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.55μs 0.904ns 3.38ns 0.0225 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 2.28μs 0.673ns 2.52ns 0.0216 0 0 1.64 KB
master EnrichedLog net472 2.61μs 2.49ns 9.32ns 0.249 0 0 1.57 KB
#5565 EnrichedLog net6.0 1.43μs 0.654ns 2.45ns 0.0229 0 0 1.64 KB
#5565 EnrichedLog netcoreapp3.1 2.39μs 1.36ns 4.71ns 0.0216 0 0 1.64 KB
#5565 EnrichedLog net472 2.71μs 2.18ns 8.43ns 0.25 0 0 1.57 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 115μs 51.7ns 200ns 0.0576 0 0 4.28 KB
master EnrichedLog netcoreapp3.1 119μs 105ns 394ns 0.0599 0 0 4.28 KB
master EnrichedLog net472 148μs 193ns 721ns 0.671 0.224 0 4.46 KB
#5565 EnrichedLog net6.0 117μs 152ns 589ns 0.0579 0 0 4.28 KB
#5565 EnrichedLog netcoreapp3.1 119μs 144ns 518ns 0 0 0 4.28 KB
#5565 EnrichedLog net472 151μs 148ns 574ns 0.68 0.227 0 4.46 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.02μs 1.11ns 4.31ns 0.0315 0 0 2.2 KB
master EnrichedLog netcoreapp3.1 4.34μs 1.22ns 4.74ns 0.0285 0 0 2.2 KB
master EnrichedLog net472 4.83μs 1.83ns 7.09ns 0.321 0 0 2.02 KB
#5565 EnrichedLog net6.0 2.95μs 0.626ns 2.42ns 0.031 0 0 2.2 KB
#5565 EnrichedLog netcoreapp3.1 4.28μs 0.942ns 3.65ns 0.0299 0 0 2.2 KB
#5565 EnrichedLog net472 5.01μs 1.09ns 4.23ns 0.32 0 0 2.02 KB
Benchmarks.Trace.RedisBenchmark - Faster :tada: Same allocations :heavy_check_mark:

Faster :tada: in #5565

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.RedisBenchmark.SendReceive‑net6.0 1.119 1,413.26 1,263.35

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.41μs 0.827ns 3.2ns 0.0163 0 0 1.14 KB
master SendReceive netcoreapp3.1 1.78μs 0.728ns 2.82ns 0.0152 0 0 1.14 KB
master SendReceive net472 2.06μs 2.05ns 7.93ns 0.183 0.00103 0 1.16 KB
#5565 SendReceive net6.0 1.26μs 1.92ns 7.42ns 0.0156 0 0 1.14 KB
#5565 SendReceive netcoreapp3.1 1.67μs 1.81ns 7.01ns 0.0157 0 0 1.14 KB
#5565 SendReceive net472 2.07μs 0.793ns 2.86ns 0.184 0 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.72μs 0.928ns 3.59ns 0.0217 0 0 1.6 KB
master EnrichedLog netcoreapp3.1 3.93μs 1.16ns 4.51ns 0.0216 0 0 1.65 KB
master EnrichedLog net472 4.33μs 3.17ns 12.3ns 0.322 0 0 2.04 KB
#5565 EnrichedLog net6.0 2.71μs 1.11ns 4.3ns 0.0218 0 0 1.6 KB
#5565 EnrichedLog netcoreapp3.1 3.95μs 1.41ns 5.47ns 0.0217 0 0 1.65 KB
#5565 EnrichedLog net472 4.44μs 3.19ns 12.3ns 0.323 0 0 2.04 KB
Benchmarks.Trace.SpanBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 419ns 0.349ns 1.35ns 0.00816 0 0 576 B
master StartFinishSpan netcoreapp3.1 544ns 0.193ns 0.668ns 0.00785 0 0 576 B
master StartFinishSpan net472 692ns 0.824ns 3.19ns 0.0917 0 0 578 B
master StartFinishScope net6.0 520ns 0.25ns 0.966ns 0.00989 0 0 696 B
master StartFinishScope netcoreapp3.1 686ns 0.719ns 2.78ns 0.0094 0 0 696 B
master StartFinishScope net472 872ns 0.597ns 2.31ns 0.105 0 0 658 B
#5565 StartFinishSpan net6.0 399ns 0.178ns 0.69ns 0.00809 0 0 576 B
#5565 StartFinishSpan netcoreapp3.1 598ns 0.162ns 0.605ns 0.00778 0 0 576 B
#5565 StartFinishSpan net472 659ns 0.173ns 0.623ns 0.0917 0 0 578 B
#5565 StartFinishScope net6.0 494ns 0.175ns 0.676ns 0.00985 0 0 696 B
#5565 StartFinishScope netcoreapp3.1 760ns 0.254ns 0.983ns 0.00936 0 0 696 B
#5565 StartFinishScope net472 835ns 0.313ns 1.21ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Slower :warning: Same allocations :heavy_check_mark:

Slower :warning: in #5565

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0 1.138 588.78 670.03

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 589ns 0.218ns 0.843ns 0.00969 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 860ns 2.01ns 7.79ns 0.00948 0 0 696 B
master RunOnMethodBegin net472 1.12μs 0.94ns 3.64ns 0.104 0 0 658 B
#5565 RunOnMethodBegin net6.0 670ns 0.176ns 0.681ns 0.01 0 0 696 B
#5565 RunOnMethodBegin netcoreapp3.1 854ns 0.406ns 1.57ns 0.00936 0 0 696 B
#5565 RunOnMethodBegin net472 1.08μs 0.336ns 1.3ns 0.104 0 0 658 B

andrewlock avatar May 15 '24 16:05 andrewlock

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5565) (11.682M)   : 0, 11682146
    master (11.565M)   : 0, 11564972
    benchmarks/2.9.0 (11.530M)   : 0, 11530110

    section Automatic
    This PR (5565) (7.851M)   : 0, 7851314
    master (7.786M)   : 0, 7786143
    benchmarks/2.9.0 (8.205M)   : 0, 8204760

    section Trace stats
    master (8.122M)   : 0, 8121929

    section Manual
    This PR (5565) (10.013M)   : 0, 10012706
    master (9.839M)   : 0, 9838575

    section Manual + Automatic
    This PR (5565) (7.439M)   : 0, 7439146
    master (7.385M)   : 0, 7384651

    section Version Conflict
    master (6.551M)   : 0, 6550532


gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5565) (9.582M)   : 0, 9582246
    master (9.828M)   : 0, 9827944

    section Automatic
    This PR (5565) (6.525M)   : 0, 6524544
    master (6.627M)   : 0, 6626617

    section Trace stats
    master (6.860M)   : 0, 6860409

    section Manual
    This PR (5565) (8.248M)   : 0, 8247538
    master (8.260M)   : 0, 8260372

    section Manual + Automatic
    This PR (5565) (6.062M)   : 0, 6061938
    master (6.131M)   : 0, 6131058

    section Version Conflict
    master (5.603M)   : 0, 5603280


gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5565) (10.118M)   : 0, 10118429
    benchmarks/2.9.0 (10.176M)   : 0, 10176148

    section Automatic
    This PR (5565) (7.211M)   : 0, 7210753
    benchmarks/2.9.0 (7.354M)   : 0, 7353635

    section Manual
    This PR (5565) (8.910M)   : 0, 8909668

    section Manual + Automatic
    This PR (5565) (6.917M)   : 0, 6916653

andrewlock avatar May 15 '24 16:05 andrewlock

Benchmarks Report for appsec :snail:

Benchmarks for #5565 compared to master:

  • All benchmarks have the same speed
  • 3 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 73μs 60.7ns 235ns 0.0728 0 0 6 KB
master AllCycleSimpleBody netcoreapp3.1 62.2μs 63.3ns 245ns 0.093 0 0 6.95 KB
master AllCycleSimpleBody net472 48.8μs 25.4ns 91.5ns 1.32 0 0 8.34 KB
master AllCycleMoreComplexBody net6.0 77.9μs 47.4ns 184ns 0.117 0 0 9.51 KB
master AllCycleMoreComplexBody netcoreapp3.1 69.4μs 75.8ns 294ns 0.139 0 0 10.37 KB
master AllCycleMoreComplexBody net472 56.1μs 52.9ns 205ns 1.87 0.0279 0 11.85 KB
master ObjectExtractorSimpleBody net6.0 139ns 0.123ns 0.46ns 0.00393 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 198ns 0.134ns 0.484ns 0.0038 0 0 272 B
master ObjectExtractorSimpleBody net472 164ns 0.287ns 1.11ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 2.98μs 1.24ns 4.65ns 0.0536 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 3.93μs 1.55ns 5.58ns 0.0511 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 4.05μs 1.54ns 5.77ns 0.603 0.00611 0 3.8 KB
#5565 AllCycleSimpleBody net6.0 73.2μs 130ns 505ns 0.0738 0 0 6.01 KB
#5565 AllCycleSimpleBody netcoreapp3.1 62.6μs 86.2ns 322ns 0.0934 0 0 6.95 KB
#5565 AllCycleSimpleBody net472 48.4μs 72ns 279ns 1.31 0 0 8.34 KB
#5565 AllCycleMoreComplexBody net6.0 78.4μs 89.9ns 348ns 0.116 0 0 9.51 KB
#5565 AllCycleMoreComplexBody netcoreapp3.1 69.5μs 67.9ns 254ns 0.139 0 0 10.37 KB
#5565 AllCycleMoreComplexBody net472 55.6μs 36.8ns 142ns 1.86 0.0277 0 11.85 KB
#5565 ObjectExtractorSimpleBody net6.0 145ns 0.257ns 0.993ns 0.00391 0 0 280 B
#5565 ObjectExtractorSimpleBody netcoreapp3.1 197ns 0.146ns 0.545ns 0.0037 0 0 272 B
#5565 ObjectExtractorSimpleBody net472 166ns 0.284ns 1.1ns 0.0446 0 0 281 B
#5565 ObjectExtractorMoreComplexBody net6.0 3.04μs 1.88ns 7.3ns 0.0532 0 0 3.78 KB
#5565 ObjectExtractorMoreComplexBody netcoreapp3.1 4μs 1.96ns 6.79ns 0.05 0 0 3.69 KB
#5565 ObjectExtractorMoreComplexBody net472 4.12μs 2.61ns 9.76ns 0.602 0.00619 0 3.8 KB
Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EncodeArgs net6.0 37.2μs 21.5ns 77.5ns 0.466 0 0 32.4 KB
master EncodeArgs netcoreapp3.1 54μs 27.4ns 106ns 0.43 0 0 32.4 KB
master EncodeArgs net472 65.7μs 38.9ns 151ns 5.15 0.0657 0 32.5 KB
master EncodeLegacyArgs net6.0 71.6μs 28.8ns 99.6ns 0 0 0 2.14 KB
master EncodeLegacyArgs netcoreapp3.1 109μs 96.7ns 374ns 0 0 0 2.14 KB
master EncodeLegacyArgs net472 151μs 68.4ns 265ns 0.302 0 0 2.15 KB
#5565 EncodeArgs net6.0 37.6μs 37.8ns 147ns 0.449 0 0 32.4 KB
#5565 EncodeArgs netcoreapp3.1 53.6μs 26.9ns 104ns 0.455 0 0 32.4 KB
#5565 EncodeArgs net472 64.7μs 35ns 131ns 5.15 0.0644 0 32.5 KB
#5565 EncodeLegacyArgs net6.0 78.3μs 34.8ns 135ns 0 0 0 2.14 KB
#5565 EncodeLegacyArgs netcoreapp3.1 106μs 263ns 1.02μs 0 0 0 2.14 KB
#5565 EncodeLegacyArgs net472 153μs 97.5ns 378ns 0.305 0 0 2.15 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWafRealisticBenchmark net6.0 182μs 124ns 479ns 0 0 0 2.42 KB
master RunWafRealisticBenchmark netcoreapp3.1 194μs 290ns 1.12μs 0 0 0 2.37 KB
master RunWafRealisticBenchmark net472 206μs 66ns 247ns 0.309 0 0 2.43 KB
master RunWafRealisticBenchmarkWithAttack net6.0 120μs 118ns 427ns 0 0 0 1.46 KB
master RunWafRealisticBenchmarkWithAttack netcoreapp3.1 130μs 91.7ns 355ns 0 0 0 1.45 KB
master RunWafRealisticBenchmarkWithAttack net472 137μs 49.3ns 191ns 0.205 0 0 1.48 KB
#5565 RunWafRealisticBenchmark net6.0 181μs 55.5ns 208ns 0 0 0 2.42 KB
#5565 RunWafRealisticBenchmark netcoreapp3.1 193μs 130ns 488ns 0 0 0 2.37 KB
#5565 RunWafRealisticBenchmark net472 205μs 60.2ns 233ns 0.308 0 0 2.43 KB
#5565 RunWafRealisticBenchmarkWithAttack net6.0 121μs 65.8ns 237ns 0 0 0 1.46 KB
#5565 RunWafRealisticBenchmarkWithAttack netcoreapp3.1 129μs 98.9ns 343ns 0 0 0 1.45 KB
#5565 RunWafRealisticBenchmarkWithAttack net472 139μs 29.7ns 115ns 0.208 0 0 1.48 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed :heavy_check_mark: More allocations :warning:

More allocations :warning: in #5565

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472 57.9 KB 59.22 KB 1.33 KB 2.29%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1 254.2 KB 256.56 KB 2.36 KB 0.93%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net6.0 254.26 KB 255.79 KB 1.53 KB 0.60%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark net6.0 59.3μs 634ns 6.04μs 0 0 0 43.44 KB
master StringConcatBenchmark netcoreapp3.1 62.8μs 727ns 7.01μs 0 0 0 42.64 KB
master StringConcatBenchmark net472 37.3μs 96ns 359ns 0 0 0 57.9 KB
master StringConcatAspectBenchmark net6.0 299μs 1.38μs 8.7μs 0 0 0 254.26 KB
master StringConcatAspectBenchmark netcoreapp3.1 333μs 1.82μs 10.5μs 0 0 0 254.2 KB
master StringConcatAspectBenchmark net472 301μs 7.35μs 72.4μs 0 0 0 278.53 KB
#5565 StringConcatBenchmark net6.0 58.4μs 694ns 6.83μs 0 0 0 43.44 KB
#5565 StringConcatBenchmark netcoreapp3.1 61.3μs 734ns 7.3μs 0 0 0 42.64 KB
#5565 StringConcatBenchmark net472 37.7μs 161ns 601ns 0 0 0 59.22 KB
#5565 StringConcatAspectBenchmark net6.0 303μs 1.66μs 10.1μs 0 0 0 255.79 KB
#5565 StringConcatAspectBenchmark netcoreapp3.1 354μs 1.74μs 7.57μs 0 0 0 256.56 KB
#5565 StringConcatAspectBenchmark net472 292μs 5.91μs 57.6μs 0 0 0 278.53 KB

andrewlock avatar May 23 '24 11:05 andrewlock

Snapshots difference summary

The following differences have been observed in committed snapshots. It is meant to help the reviewer. The diff is simplistic, so please check some files anyway while we improve it.

169 occurrences of :

-      _sampling_priority_v1: 1.0
+      _sampling_priority_v1: 2.0

github-actions[bot] avatar Jun 05 '24 12:06 github-actions[bot]

Datadog Report

Branch report: flavien/asm/standalone-billing Commit report: 5b8f8fa Test service: dd-trace-dotnet

:white_check_mark: 0 Failed, 435249 Passed, 2688 Skipped, 26h 30m 28.22s Total Time :snowflake: 2 New Flaky

New Flaky Tests (2)

  • NoExceptions - Datadog.Trace.ClrProfiler.IntegrationTests.SmokeTests.RuntimeMetricsShutdownSmokeTest - Last Failure

    Expand for error 
    xpected no errors in smoke test: Fatal error. 0xC0000005
  at System.Diagnostics.Tracing.EventPipePayloadDecoder.DecodePayload(EventMetadata ByRef, System.ReadOnlySpan\`1<Byte>)
  at System.Diagnostics.Tracing.EventPipeEventDispatcher.DispatchEventsToEventListeners()
  at System.Threading.Tasks.Task.InnerInvoke()
  at System.Threading.Tasks.Task+<>c.<.cctor>b__272_0(System.Object)
  at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
  at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
  at System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread)
  at System.Threading.Tasks.ThreadPoolTaskScheduler+<>c.<.cctor>b__10_0(System.Object)
  at System.Threading.Thread.StartCallback()
..

  • NoExceptions - Datadog.Trace.ClrProfiler.IntegrationTests.SmokeTests.RuntimeMetricsShutdownSmokeTest - Last Failure

    Expand for error 
    xpected exit code: 0, actual exit code: 134. Message: Fatal error. System.AccessViolationException: Attempted to read or write protected memory. This is often an indication that other memory is corrupt.
  at System.Diagnostics.Tracing.EventPipePayloadDecoder.DecodePayload(EventMetadata ByRef, System.ReadOnlySpan\`1<Byte>)
  at System.Diagnostics.Tracing.NativeRuntimeEventSource.ProcessEvent(UInt32, UInt32, System.DateTime, System.Guid, System.Guid, System.ReadOnlySpan\`1<Byte>)
  at System.Diagnostics.Tracing.EventPipeEventDispatcher.DispatchEventsToEventListeners()
  at System.Threading.Tasks.Task.InnerInvoke()
  at System.Threading.Tasks.Task+<>c.<.cctor>b__272_0(System.Object)
  at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
  at System.Threading.Tasks.Task.ExecuteWithThreadLocal(System.Threading.Tasks.Task ByRef, System.Threading.Thread)
  at System.Threading.Tasks.Task.ExecuteEntryUnsafe(System.Threading.Thread)
  at System.Threading.Tasks.ThreadPoolTaskScheduler+<>c.<.cctor>b__10_0(System.Object)
..

datadog-ddstaging[bot] avatar Jul 12 '24 15:07 datadog-ddstaging[bot]