dd-trace-dotnet icon indicating copy to clipboard operation
dd-trace-dotnet copied to clipboard

[ASM] Added telemetry and limits to WAF Unmanaged Pool

Open daniel-romano-DD opened this issue 1 year ago • 7 comments

Summary of changes

Added telemetry to survey the status of the new WAF Unmanaged Pools (number of "pooled" and "slow" pools) Added a limit to the amount of "pooled" or "fast" unmanaged memory pools, to limit the case where the number of threads serving request goes out of control.

Reason for change

The change in the WAF encoder is quite big and we've had some crashes that made us disable it. Now we will have info about the system in case it crashes again. Also, as the unmanaged pool is thread local, exists the risk that the number of threads serving a request rises, causing a possible excessive use of memory.

Implementation details

Added two gauges: waf.pool_count and waf.pool_slow_count Added a memory pool factory delivering a limited number of "fast" pools and when it has been surpassed, slow ones.

daniel-romano-DD avatar Feb 07 '24 10:02 daniel-romano-DD

Datadog Report

Branch report: dani/asm/pool_telemetry Commit report: 0bf14ee Test service: dd-trace-dotnet

:x: 1 Failed (0 Known Flaky), 303879 Passed, 1571 Skipped, 39m 25.96s Wall Time

:x: Failed Tests (1)

  • TelemetryControllerDumpsAllTelemetryToFile - Datadog.Trace.Tests.Telemetry.TelemetryControllerTests - Details

    Expand for error
    xpected object not to be <null>.
    

datadog-ddstaging[bot] avatar Feb 07 '24 10:02 datadog-ddstaging[bot]

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5157) - mean (72ms)  : 64, 80
     .   : milestone, 72,
    master - mean (74ms)  : 66, 81
     .   : milestone, 74,

    section CallTarget+Inlining+NGEN
    This PR (5157) - mean (1,015ms)  : 996, 1035
     .   : milestone, 1015,
    master - mean (1,016ms)  : 992, 1039
     .   : milestone, 1016,

gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5157) - mean (110ms)  : 106, 113
     .   : milestone, 110,
    master - mean (109ms)  : 106, 112
     .   : milestone, 109,

    section CallTarget+Inlining+NGEN
    This PR (5157) - mean (735ms)  : 718, 752
     .   : milestone, 735,
    master - mean (737ms)  : 716, 759
     .   : milestone, 737,

gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5157) - mean (93ms)  : 90, 96
     .   : milestone, 93,
    master - mean (94ms)  : 91, 98
     .   : milestone, 94,

    section CallTarget+Inlining+NGEN
    This PR (5157) - mean (688ms)  : 672, 703
     .   : milestone, 688,
    master - mean (693ms)  : 675, 711
     .   : milestone, 693,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5157) - mean (195ms)  : 191, 198
     .   : milestone, 195,
    master - mean (192ms)  : 189, 194
     .   : milestone, 192,

    section CallTarget+Inlining+NGEN
    This PR (5157) - mean (1,097ms)  : 1078, 1116
     .   : milestone, 1097,
    master - mean (1,094ms)  : 1072, 1116
     .   : milestone, 1094,

gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5157) - mean (280ms)  : 276, 284
     .   : milestone, 280,
    master - mean (277ms)  : 272, 281
     .   : milestone, 277,

    section CallTarget+Inlining+NGEN
    This PR (5157) - mean (901ms)  : 877, 925
     .   : milestone, 901,
    master - mean (884ms)  : 859, 909
     .   : milestone, 884,

gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5157) - mean (270ms)  : 264, 276
     .   : milestone, 270,
    master - mean (265ms)  : 259, 270
     .   : milestone, 265,

    section CallTarget+Inlining+NGEN
    This PR (5157) - mean (876ms)  : 849, 903
     .   : milestone, 876,
    master - mean (865ms)  : 835, 894
     .   : milestone, 865,

andrewlock avatar Feb 07 '24 10:02 andrewlock

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5157) (11.578M)   : 0, 11577798
    master (11.522M)   : 0, 11522381
    benchmarks/2.9.0 (11.170M)   : 0, 11169786

    section Automatic
    This PR (5157) (7.891M)   : 0, 7891438
    master (7.941M)   : 0, 7940524
    benchmarks/2.9.0 (8.435M)   : 0, 8434769

    section Trace stats
    master (8.315M)   : 0, 8314530

    section Manual
    This PR (5157) (10.074M)   : 0, 10073578
    master (9.988M)   : 0, 9988179

    section Manual + Automatic
    This PR (5157) (7.479M)   : 0, 7479216
    master (7.586M)   : 0, 7586252

    section Version Conflict
    master (6.763M)   : 0, 6762646

gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5157) (9.629M)   : 0, 9629090
    master (9.577M)   : 0, 9576962
    benchmarks/2.9.0 (9.658M)   : 0, 9657706

    section Automatic
    This PR (5157) (6.595M)   : 0, 6594679
    master (6.559M)   : 0, 6558544

    section Trace stats
    master (6.899M)   : 0, 6899377

    section Manual
    This PR (5157) (8.266M)   : 0, 8266271
    master (8.285M)   : 0, 8284821

    section Manual + Automatic
    This PR (5157) (6.243M)   : 0, 6243352
    master (6.060M)   : 0, 6060400

    section Version Conflict
    master (5.577M)   : 0, 5576823

gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5157) (9.902M)   : 0, 9902313
    master (9.719M)   : 0, 9718909
    benchmarks/2.9.0 (10.068M)   : 0, 10068415

    section Automatic
    This PR (5157) (7.145M)   : 0, 7145308
    master (7.165M)   : 0, 7165128
    benchmarks/2.9.0 (7.475M)   : 0, 7475050

    section Trace stats
    master (7.388M)   : 0, 7387740

    section Manual
    This PR (5157) (8.742M)   : 0, 8741826
    master (8.762M)   : 0, 8761881

    section Manual + Automatic
    This PR (5157) (6.855M)   : 0, 6854908
    master (6.864M)   : 0, 6863897

    section Version Conflict
    master (6.179M)   : 0, 6179007

andrewlock avatar Feb 07 '24 11:02 andrewlock

Benchmarks Report :snail:

Benchmarks for #5157 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.150
  • 1 benchmarks have fewer allocations
  • 2 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 9.02μs 51.2ns 362ns 0.0222 0.00886 0 7.5 KB
master StartStopWithChild netcoreapp3.1 10.3μs 51.4ns 241ns 0.0201 0.00503 0 7.59 KB
master StartStopWithChild net472 17.1μs 71.9ns 279ns 1.31 0.323 0.102 7.97 KB
#5157 StartStopWithChild net6.0 8.91μs 48ns 284ns 0.0338 0.0169 0.00423 7.49 KB
#5157 StartStopWithChild netcoreapp3.1 11.1μs 62.6ns 415ns 0.0262 0.0105 0 7.59 KB
#5157 StartStopWithChild net472 16.8μs 67ns 259ns 1.34 0.374 0.127 7.96 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 461μs 384ns 1.49μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 617μs 311ns 1.21μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 809μs 255ns 954ns 0.406 0 0 3.3 KB
#5157 WriteAndFlushEnrichedTraces net6.0 454μs 190ns 684ns 0 0 0 2.7 KB
#5157 WriteAndFlushEnrichedTraces netcoreapp3.1 627μs 422ns 1.63μs 0 0 0 2.7 KB
#5157 WriteAndFlushEnrichedTraces net472 816μs 406ns 1.57μs 0.406 0 0 3.3 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 40.7μs 29.9ns 112ns 0.02 0 0 2.36 KB
master AllCycleSimpleBody netcoreapp3.1 43.4μs 153ns 592ns 0.0214 0 0 2.34 KB
master AllCycleSimpleBody net472 47.1μs 27.7ns 107ns 0.372 0 0 2.41 KB
master AllCycleMoreComplexBody net6.0 213μs 70.4ns 273ns 0.107 0 0 9.84 KB
master AllCycleMoreComplexBody netcoreapp3.1 226μs 192ns 742ns 0.112 0 0 9.73 KB
master AllCycleMoreComplexBody net472 242μs 98.4ns 355ns 1.57 0 0 9.91 KB
master ObjectExtractorSimpleBody net6.0 156ns 0.174ns 0.673ns 0.00393 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 199ns 0.113ns 0.422ns 0.00369 0 0 272 B
master ObjectExtractorSimpleBody net472 172ns 0.144ns 0.559ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 3.01μs 3.13ns 11.3ns 0.0525 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 4.05μs 1.59ns 5.93ns 0.0505 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 3.8μs 1.44ns 5.4ns 0.603 0.0057 0 3.8 KB
#5157 AllCycleSimpleBody net6.0 41.2μs 50.4ns 182ns 0.0204 0 0 2.36 KB
#5157 AllCycleSimpleBody netcoreapp3.1 44.5μs 94.8ns 367ns 0.0221 0 0 2.34 KB
#5157 AllCycleSimpleBody net472 47.9μs 20.6ns 79.9ns 0.362 0 0 2.41 KB
#5157 AllCycleMoreComplexBody net6.0 213μs 51.3ns 192ns 0.107 0 0 9.84 KB
#5157 AllCycleMoreComplexBody netcoreapp3.1 226μs 142ns 550ns 0.113 0 0 9.73 KB
#5157 AllCycleMoreComplexBody net472 240μs 66.7ns 250ns 1.55 0 0 9.91 KB
#5157 ObjectExtractorSimpleBody net6.0 141ns 0.116ns 0.435ns 0.00398 0 0 280 B
#5157 ObjectExtractorSimpleBody netcoreapp3.1 196ns 0.116ns 0.448ns 0.00366 0 0 272 B
#5157 ObjectExtractorSimpleBody net472 168ns 0.136ns 0.527ns 0.0446 0 0 281 B
#5157 ObjectExtractorMoreComplexBody net6.0 3.02μs 1.58ns 6.13ns 0.0527 0 0 3.78 KB
#5157 ObjectExtractorMoreComplexBody netcoreapp3.1 3.98μs 2.51ns 9.39ns 0.0499 0 0 3.69 KB
#5157 ObjectExtractorMoreComplexBody net472 4.2μs 1.91ns 7.41ns 0.602 0.0063 0 3.8 KB
Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EncodeArgs net6.0 51.9μs 51.5ns 193ns 0.418 0 0 30.94 KB
master EncodeArgs netcoreapp3.1 70.1μs 101ns 390ns 0.418 0 0 31.47 KB
master EncodeArgs net472 85μs 41.9ns 162ns 5.12 0.0846 0 32.27 KB
master EncodeLegacyArgs net6.0 131μs 311ns 1.16μs 0.487 0 0 33.89 KB
master EncodeLegacyArgs netcoreapp3.1 155μs 296ns 1.15μs 0.472 0 0 34.08 KB
master EncodeLegacyArgs net472 217μs 327ns 1.27μs 5.46 0.42 0 35 KB
#5157 EncodeArgs net6.0 51.6μs 67.1ns 251ns 0.438 0 0 30.94 KB
#5157 EncodeArgs netcoreapp3.1 69.9μs 82.8ns 321ns 0.416 0 0 31.47 KB
#5157 EncodeArgs net472 84.6μs 49.8ns 193ns 5.1 0.0849 0 32.27 KB
#5157 EncodeLegacyArgs net6.0 126μs 597ns 2.39μs 0.439 0 0 33.89 KB
#5157 EncodeLegacyArgs netcoreapp3.1 155μs 429ns 1.66μs 0.466 0 0 34.09 KB
#5157 EncodeLegacyArgs net472 219μs 698ns 2.71μs 5.46 0.42 0 35 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWafRealisticBenchmark net6.0 189μs 104ns 391ns 0.0947 0 0 6.51 KB
master RunWafRealisticBenchmark netcoreapp3.1 205μs 324ns 1.26μs 0 0 0 6.49 KB
master RunWafRealisticBenchmark net472 224μs 522ns 2.02μs 0.997 0 0 6.59 KB
master RunWafRealisticBenchmarkWithAttack net6.0 126μs 193ns 839ns 0.0635 0 0 4.15 KB
master RunWafRealisticBenchmarkWithAttack netcoreapp3.1 134μs 198ns 768ns 0 0 0 4.15 KB
master RunWafRealisticBenchmarkWithAttack net472 147μs 39.7ns 143ns 0.659 0 0 4.19 KB
#5157 RunWafRealisticBenchmark net6.0 196μs 84.2ns 315ns 0.097 0 0 6.51 KB
#5157 RunWafRealisticBenchmark netcoreapp3.1 207μs 483ns 1.87μs 0 0 0 6.49 KB
#5157 RunWafRealisticBenchmark net472 224μs 71ns 256ns 1.01 0 0 6.59 KB
#5157 RunWafRealisticBenchmarkWithAttack net6.0 127μs 294ns 1.1μs 0.0635 0 0 4.15 KB
#5157 RunWafRealisticBenchmarkWithAttack netcoreapp3.1 135μs 172ns 668ns 0 0 0 4.14 KB
#5157 RunWafRealisticBenchmarkWithAttack net472 146μs 38.5ns 139ns 0.657 0 0 4.19 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 170μs 194ns 750ns 0.253 0 0 18.26 KB
master SendRequest netcoreapp3.1 188μs 150ns 563ns 0.187 0 0 20.42 KB
master SendRequest net472 0.000841ns 0.000381ns 0.00143ns 0 0 0 0 b
#5157 SendRequest net6.0 169μs 191ns 714ns 0.252 0 0 18.26 KB
#5157 SendRequest netcoreapp3.1 191μs 215ns 834ns 0.191 0 0 20.42 KB
#5157 SendRequest net472 0.0011ns 0.000337ns 0.0013ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: Fewer allocations :tada:

Fewer allocations :tada: in #5157

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0 41.86 KB 41.43 KB -431 B -1.03%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 548μs 1.46μs 5.65μs 0.548 0 0 41.86 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 650μs 1.85μs 7.15μs 0.324 0 0 41.69 KB
master WriteAndFlushEnrichedTraces net472 833μs 4.15μs 18.1μs 8.17 2.45 0.408 53.25 KB
#5157 WriteAndFlushEnrichedTraces net6.0 533μs 305ns 1.1μs 0.53 0 0 41.43 KB
#5157 WriteAndFlushEnrichedTraces netcoreapp3.1 651μs 1.04μs 4.03μs 0.326 0 0 41.73 KB
#5157 WriteAndFlushEnrichedTraces net472 838μs 3.93μs 15.2μs 8.22 2.47 0.411 53.25 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.12μs 0.927ns 3.59ns 0.0106 0 0 776 B
master ExecuteNonQuery netcoreapp3.1 1.49μs 1.21ns 4.67ns 0.0105 0 0 776 B
master ExecuteNonQuery net472 1.73μs 0.999ns 3.87ns 0.117 0 0 738 B
#5157 ExecuteNonQuery net6.0 1.12μs 1.59ns 6.15ns 0.0107 0 0 776 B
#5157 ExecuteNonQuery netcoreapp3.1 1.56μs 0.419ns 1.62ns 0.0101 0 0 776 B
#5157 ExecuteNonQuery net472 1.73μs 0.822ns 3.07ns 0.117 0 0 738 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.28μs 0.669ns 2.5ns 0.0128 0 0 944 B
master CallElasticsearch netcoreapp3.1 1.56μs 1.48ns 5.14ns 0.0126 0 0 944 B
master CallElasticsearch net472 2.59μs 1.4ns 5.25ns 0.153 0 0 963 B
master CallElasticsearchAsync net6.0 1.25μs 0.757ns 2.83ns 0.013 0 0 920 B
master CallElasticsearchAsync netcoreapp3.1 1.59μs 0.519ns 1.87ns 0.0135 0 0 992 B
master CallElasticsearchAsync net472 2.61μs 0.814ns 3.15ns 0.162 0 0 1.02 KB
#5157 CallElasticsearch net6.0 1.23μs 0.511ns 1.91ns 0.0129 0 0 944 B
#5157 CallElasticsearch netcoreapp3.1 1.55μs 0.803ns 3.11ns 0.0124 0 0 944 B
#5157 CallElasticsearch net472 2.58μs 0.783ns 3.03ns 0.153 0 0 963 B
#5157 CallElasticsearchAsync net6.0 1.36μs 0.444ns 1.66ns 0.0129 0 0 920 B
#5157 CallElasticsearchAsync netcoreapp3.1 1.68μs 2.31ns 8.66ns 0.0134 0 0 992 B
#5157 CallElasticsearchAsync net472 2.59μs 0.809ns 3.03ns 0.161 0 0 1.02 KB
Benchmarks.Trace.GraphQLBenchmark - Faster :tada: Same allocations :heavy_check_mark:

Faster :tada: in #5157

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑net472 1.150 1,954.64 1,699.56

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.27μs 0.617ns 2.39ns 0.0127 0 0 920 B
master ExecuteAsync netcoreapp3.1 1.71μs 1.15ns 4.31ns 0.0121 0 0 920 B
master ExecuteAsync net472 1.95μs 0.821ns 3.18ns 0.139 0 0 883 B
#5157 ExecuteAsync net6.0 1.32μs 0.528ns 1.97ns 0.0125 0 0 920 B
#5157 ExecuteAsync netcoreapp3.1 1.63μs 0.845ns 3.27ns 0.0122 0 0 920 B
#5157 ExecuteAsync net472 1.7μs 0.43ns 1.55ns 0.14 0 0 883 B
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 3.98μs 2.34ns 8.45ns 0.0279 0 0 2.1 KB
master SendAsync netcoreapp3.1 4.91μs 1.54ns 5.78ns 0.0345 0 0 2.64 KB
master SendAsync net472 7.75μs 1.59ns 6.17ns 0.526 0 0 3.31 KB
#5157 SendAsync net6.0 4.03μs 2.04ns 7.89ns 0.0282 0 0 2.1 KB
#5157 SendAsync netcoreapp3.1 5.02μs 2.72ns 10.5ns 0.0351 0 0 2.64 KB
#5157 SendAsync net472 7.78μs 2.63ns 10.2ns 0.524 0 0 3.31 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed :heavy_check_mark: More allocations :warning:

More allocations :warning: in #5157

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472 55.75 KB 61.77 KB 6.02 KB 10.79%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑net472 221.18 KB 229.38 KB 8.19 KB 3.70%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark net6.0 51.1μs 208ns 749ns 0 0 0 43.44 KB
master StringConcatBenchmark netcoreapp3.1 61μs 869ns 8.56μs 0 0 0 42.64 KB
master StringConcatBenchmark net472 37.3μs 67.1ns 242ns 0 0 0 55.75 KB
master StringConcatAspectBenchmark net6.0 271μs 2.98μs 27.5μs 0 0 0 204.86 KB
master StringConcatAspectBenchmark netcoreapp3.1 277μs 1.17μs 5.46μs 0 0 0 201.66 KB
master StringConcatAspectBenchmark net472 233μs 1.23μs 7.19μs 0 0 0 221.18 KB
#5157 StringConcatBenchmark net6.0 56.1μs 711ns 6.97μs 0 0 0 43.44 KB
#5157 StringConcatBenchmark netcoreapp3.1 52.6μs 263ns 1.08μs 0 0 0 42.64 KB
#5157 StringConcatBenchmark net472 37.1μs 153ns 551ns 0 0 0 61.77 KB
#5157 StringConcatAspectBenchmark net6.0 274μs 1.35μs 6.76μs 0 0 0 204.03 KB
#5157 StringConcatAspectBenchmark netcoreapp3.1 282μs 1.29μs 8.48μs 0 0 0 201.49 KB
#5157 StringConcatAspectBenchmark net472 253μs 4.95μs 48.5μs 0 0 0 229.38 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.52μs 0.832ns 3.11ns 0.022 0 0 1.58 KB
master EnrichedLog netcoreapp3.1 2.27μs 1.08ns 4.03ns 0.0214 0 0 1.58 KB
master EnrichedLog net472 2.47μs 4.55ns 17.6ns 0.239 0 0 1.51 KB
#5157 EnrichedLog net6.0 1.44μs 0.586ns 2.27ns 0.0219 0 0 1.58 KB
#5157 EnrichedLog netcoreapp3.1 2.12μs 0.644ns 2.5ns 0.0211 0 0 1.58 KB
#5157 EnrichedLog net472 2.62μs 1.59ns 5.73ns 0.239 0 0 1.51 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 113μs 86.7ns 324ns 0.0564 0 0 4.22 KB
master EnrichedLog netcoreapp3.1 118μs 126ns 488ns 0.0592 0 0 4.22 KB
master EnrichedLog net472 147μs 52ns 202ns 0.659 0.22 0 4.4 KB
#5157 EnrichedLog net6.0 114μs 81.7ns 306ns 0.0567 0 0 4.22 KB
#5157 EnrichedLog netcoreapp3.1 118μs 146ns 565ns 0.0589 0 0 4.22 KB
#5157 EnrichedLog net472 150μs 85ns 329ns 0.671 0.224 0 4.4 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.98μs 1ns 3.76ns 0.0298 0 0 2.14 KB
master EnrichedLog netcoreapp3.1 4.43μs 17.8ns 69.1ns 0.0277 0 0 2.14 KB
master EnrichedLog net472 4.96μs 0.882ns 3.42ns 0.309 0 0 1.95 KB
#5157 EnrichedLog net6.0 2.95μs 1ns 3.89ns 0.0295 0 0 2.14 KB
#5157 EnrichedLog netcoreapp3.1 4.16μs 2.8ns 10.9ns 0.029 0 0 2.14 KB
#5157 EnrichedLog net472 4.7μs 1.48ns 5.54ns 0.31 0 0 1.95 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.34μs 0.237ns 0.856ns 0.0154 0 0 1.11 KB
master SendReceive netcoreapp3.1 1.81μs 1.5ns 5.8ns 0.0153 0 0 1.11 KB
master SendReceive net472 2.17μs 3.06ns 11.9ns 0.178 0 0 1.12 KB
#5157 SendReceive net6.0 1.3μs 0.471ns 1.83ns 0.0158 0 0 1.11 KB
#5157 SendReceive netcoreapp3.1 1.84μs 0.964ns 3.73ns 0.0146 0 0 1.11 KB
#5157 SendReceive net472 2.12μs 0.998ns 3.6ns 0.178 0 0 1.12 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.85μs 0.725ns 2.71ns 0.0215 0 0 1.54 KB
master EnrichedLog netcoreapp3.1 3.92μs 1.14ns 4.26ns 0.0214 0 0 1.58 KB
master EnrichedLog net472 4.26μs 2.28ns 8.85ns 0.313 0 0 1.97 KB
#5157 EnrichedLog net6.0 2.79μs 1.29ns 4.98ns 0.021 0 0 1.54 KB
#5157 EnrichedLog netcoreapp3.1 3.97μs 3.54ns 13.7ns 0.0218 0 0 1.58 KB
#5157 EnrichedLog net472 4.15μs 2.24ns 8.4ns 0.312 0 0 1.97 KB
Benchmarks.Trace.SpanBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 501ns 0.462ns 1.79ns 0.0076 0 0 544 B
master StartFinishSpan netcoreapp3.1 732ns 0.808ns 3.02ns 0.00728 0 0 544 B
master StartFinishSpan net472 760ns 1.33ns 5.15ns 0.0866 0 0 546 B
master StartFinishScope net6.0 621ns 0.7ns 2.71ns 0.00928 0 0 664 B
master StartFinishScope netcoreapp3.1 809ns 0.389ns 1.51ns 0.00893 0 0 664 B
master StartFinishScope net472 992ns 1.72ns 6.66ns 0.099 0 0 626 B
#5157 StartFinishSpan net6.0 467ns 0.604ns 2.34ns 0.00756 0 0 544 B
#5157 StartFinishSpan netcoreapp3.1 681ns 1.11ns 4.02ns 0.00728 0 0 544 B
#5157 StartFinishSpan net472 711ns 1.54ns 5.32ns 0.0867 0 0 546 B
#5157 StartFinishScope net6.0 637ns 0.648ns 2.51ns 0.0093 0 0 664 B
#5157 StartFinishScope netcoreapp3.1 778ns 1.52ns 5.88ns 0.00898 0 0 664 B
#5157 StartFinishScope net472 977ns 1.95ns 7.55ns 0.0995 0 0 626 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 655ns 0.975ns 3.78ns 0.00916 0 0 664 B
master RunOnMethodBegin netcoreapp3.1 1.03μs 1.2ns 4.64ns 0.00898 0 0 664 B
master RunOnMethodBegin net472 1.08μs 1.71ns 6.62ns 0.0994 0 0 626 B
#5157 RunOnMethodBegin net6.0 650ns 0.692ns 2.68ns 0.00933 0 0 664 B
#5157 RunOnMethodBegin netcoreapp3.1 933ns 1.66ns 6.42ns 0.00911 0 0 664 B
#5157 RunOnMethodBegin net472 1.09μs 1.69ns 6.33ns 0.0994 0 0 626 B

andrewlock avatar Feb 07 '24 12:02 andrewlock

Datadog Report

Branch report: dani/asm/pool_telemetry Commit report: 265c54f Test service: dd-trace-dotnet

:white_check_mark: 0 Failed, 21510 Passed, 653 Skipped, 49m 22.58s Wall Time :hourglass: 3 Performance Regressions

:hourglass: Performance Regressions vs Default Branch (3)

  • ObjectExtractorMoreComplexBody - Benchmarks.Trace.Asm.AppSecBodyBenchmark 4.6µs (+596ns, +15%) - Details
  • Baseline - Samples.HttpMessageHandler.windows.net60.json.scenarios 270.17ms (+9.84ms, +4%) - Details
  • Baseline - Samples.HttpMessageHandler.windows.net462.json.scenarios 194.57ms (+6.67ms, +4%) - Details

datadog-ddstaging[bot] avatar Apr 17 '24 10:04 datadog-ddstaging[bot]

Benchmarks Report for tracer :snail:

Benchmarks for #5157 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.125
  • 2 benchmarks are slower, with geometric mean 1.135
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.95μs 44.1ns 225ns 0.0284 0.0142 0.00473 7.54 KB
master StartStopWithChild netcoreapp3.1 10.7μs 56.2ns 276ns 0.0253 0.0101 0 7.64 KB
master StartStopWithChild net472 17.1μs 49.2ns 191ns 1.34 0.335 0.103 8.07 KB
#5157 StartStopWithChild net6.0 8.91μs 49.4ns 292ns 0.0216 0.00863 0 7.54 KB
#5157 StartStopWithChild netcoreapp3.1 10.7μs 54.2ns 271ns 0.0266 0.0106 0 7.63 KB
#5157 StartStopWithChild net472 17.1μs 61.4ns 238ns 1.36 0.368 0.12 8.07 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 486μs 325ns 1.26μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 621μs 201ns 777ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 815μs 393ns 1.47μs 0.406 0 0 3.3 KB
#5157 WriteAndFlushEnrichedTraces net6.0 465μs 405ns 1.51μs 0 0 0 2.7 KB
#5157 WriteAndFlushEnrichedTraces netcoreapp3.1 627μs 164ns 613ns 0 0 0 2.7 KB
#5157 WriteAndFlushEnrichedTraces net472 826μs 546ns 2.12μs 0.414 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 169μs 157ns 609ns 0.252 0 0 18.38 KB
master SendRequest netcoreapp3.1 191μs 248ns 962ns 0.192 0 0 20.54 KB
master SendRequest net472 0.000134ns 0.000121ns 0.000452ns 0 0 0 0 b
#5157 SendRequest net6.0 169μs 191ns 689ns 0.254 0 0 18.38 KB
#5157 SendRequest netcoreapp3.1 190μs 334ns 1.29μs 0.19 0 0 20.54 KB
#5157 SendRequest net472 0.00105ns 0.00035ns 0.00136ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 533μs 262ns 1.02μs 0.556 0 0 41.53 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 642μs 766ns 2.97μs 0.326 0 0 41.68 KB
master WriteAndFlushEnrichedTraces net472 868μs 2.94μs 11.4μs 8.09 2.25 0.45 53.26 KB
#5157 WriteAndFlushEnrichedTraces net6.0 546μs 2.72μs 11.2μs 0.553 0 0 41.48 KB
#5157 WriteAndFlushEnrichedTraces netcoreapp3.1 657μs 964ns 3.61μs 0.331 0 0 41.75 KB
#5157 WriteAndFlushEnrichedTraces net472 870μs 4.16μs 16.1μs 8.08 2.55 0.425 53.26 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.17μs 0.429ns 1.66ns 0.0112 0 0 800 B
master ExecuteNonQuery netcoreapp3.1 1.51μs 0.877ns 3.4ns 0.0105 0 0 800 B
master ExecuteNonQuery net472 1.82μs 0.524ns 2.03ns 0.12 0 0 762 B
#5157 ExecuteNonQuery net6.0 1.16μs 0.646ns 2.5ns 0.0111 0 0 800 B
#5157 ExecuteNonQuery netcoreapp3.1 1.63μs 0.621ns 2.32ns 0.0106 0 0 800 B
#5157 ExecuteNonQuery net472 1.84μs 1.03ns 3.97ns 0.12 0 0 762 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.32μs 0.505ns 1.95ns 0.0144 0 0 1.02 KB
master CallElasticsearch netcoreapp3.1 1.59μs 0.873ns 3.27ns 0.0143 0 0 1.02 KB
master CallElasticsearch net472 2.46μs 1.49ns 5.79ns 0.164 0 0 1.04 KB
master CallElasticsearchAsync net6.0 1.29μs 0.989ns 3.7ns 0.0135 0 0 1 KB
master CallElasticsearchAsync netcoreapp3.1 1.77μs 0.811ns 3.03ns 0.0142 0 0 1.07 KB
master CallElasticsearchAsync net472 2.69μs 1.98ns 7.68ns 0.172 0 0 1.09 KB
#5157 CallElasticsearch net6.0 1.22μs 0.417ns 1.56ns 0.0146 0 0 1.02 KB
#5157 CallElasticsearch netcoreapp3.1 1.73μs 1.21ns 4.51ns 0.0138 0 0 1.02 KB
#5157 CallElasticsearch net472 2.5μs 1.42ns 5.49ns 0.164 0 0 1.04 KB
#5157 CallElasticsearchAsync net6.0 1.35μs 0.547ns 2.12ns 0.0142 0 0 1 KB
#5157 CallElasticsearchAsync netcoreapp3.1 1.63μs 1.08ns 4.04ns 0.0148 0 0 1.07 KB
#5157 CallElasticsearchAsync net472 2.63μs 1.81ns 6.78ns 0.173 0 0 1.09 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.45μs 0.509ns 1.97ns 0.013 0 0 944 B
master ExecuteAsync netcoreapp3.1 1.61μs 2.43ns 9.42ns 0.0122 0 0 944 B
master ExecuteAsync net472 1.85μs 2.88ns 11.2ns 0.143 0 0 907 B
#5157 ExecuteAsync net6.0 1.31μs 0.637ns 2.47ns 0.0132 0 0 944 B
#5157 ExecuteAsync netcoreapp3.1 1.67μs 0.921ns 3.45ns 0.0126 0 0 944 B
#5157 ExecuteAsync net472 1.81μs 1.47ns 5.52ns 0.144 0 0 907 B
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.15μs 1.71ns 6.4ns 0.031 0 0 2.18 KB
master SendAsync netcoreapp3.1 5.14μs 12.4ns 46.2ns 0.0358 0 0 2.71 KB
master SendAsync net472 7.71μs 2.44ns 9.46ns 0.484 0 0 3.07 KB
#5157 SendAsync net6.0 4.05μs 2.41ns 9ns 0.0305 0 0 2.18 KB
#5157 SendAsync netcoreapp3.1 4.97μs 1.94ns 7.26ns 0.0374 0 0 2.71 KB
#5157 SendAsync net472 7.67μs 2.23ns 8.04ns 0.486 0.00383 0 3.07 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.59μs 0.766ns 2.97ns 0.0238 0 0 1.68 KB
master EnrichedLog netcoreapp3.1 2.4μs 0.911ns 3.28ns 0.0225 0 0 1.68 KB
master EnrichedLog net472 2.81μs 3.79ns 14.7ns 0.254 0 0 1.61 KB
#5157 EnrichedLog net6.0 1.46μs 0.504ns 1.95ns 0.0235 0 0 1.68 KB
#5157 EnrichedLog netcoreapp3.1 2.39μs 1.75ns 6.55ns 0.0226 0 0 1.68 KB
#5157 EnrichedLog net472 2.76μs 3.09ns 12ns 0.254 0 0 1.61 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 113μs 149ns 575ns 0.0562 0 0 4.26 KB
master EnrichedLog netcoreapp3.1 118μs 210ns 814ns 0.0592 0 0 4.26 KB
master EnrichedLog net472 149μs 145ns 563ns 0.668 0.223 0 4.45 KB
#5157 EnrichedLog net6.0 114μs 140ns 542ns 0.0571 0 0 4.26 KB
#5157 EnrichedLog netcoreapp3.1 119μs 211ns 819ns 0 0 0 4.26 KB
#5157 EnrichedLog net472 152μs 299ns 1.16μs 0.678 0.226 0 4.45 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.93μs 0.998ns 3.74ns 0.0309 0 0 2.24 KB
master EnrichedLog netcoreapp3.1 4.19μs 1.3ns 5.02ns 0.0294 0 0 2.24 KB
master EnrichedLog net472 4.9μs 1.9ns 7.35ns 0.325 0 0 2.05 KB
#5157 EnrichedLog net6.0 3.08μs 1.37ns 5.14ns 0.0308 0 0 2.24 KB
#5157 EnrichedLog netcoreapp3.1 4.26μs 1.49ns 5.57ns 0.0297 0 0 2.24 KB
#5157 EnrichedLog net472 4.93μs 2.78ns 10.8ns 0.325 0 0 2.05 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.4μs 1.34ns 5.02ns 0.0168 0 0 1.19 KB
master SendReceive netcoreapp3.1 1.81μs 2.17ns 8.42ns 0.0163 0 0 1.19 KB
master SendReceive net472 2.2μs 1.72ns 6.2ns 0.19 0.00109 0 1.2 KB
#5157 SendReceive net6.0 1.45μs 0.639ns 2.21ns 0.0168 0 0 1.19 KB
#5157 SendReceive netcoreapp3.1 1.82μs 1.6ns 6ns 0.0163 0 0 1.19 KB
#5157 SendReceive net472 2.26μs 1.33ns 4.97ns 0.19 0 0 1.2 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.8μs 0.611ns 2.2ns 0.0224 0 0 1.58 KB
master EnrichedLog netcoreapp3.1 3.83μs 2.55ns 9.87ns 0.0212 0 0 1.63 KB
master EnrichedLog net472 4.41μs 2.56ns 9.93ns 0.319 0 0 2.02 KB
#5157 EnrichedLog net6.0 2.72μs 2.13ns 7.97ns 0.0215 0 0 1.58 KB
#5157 EnrichedLog netcoreapp3.1 4.04μs 5.18ns 20.1ns 0.0222 0 0 1.63 KB
#5157 EnrichedLog net472 4.41μs 2.15ns 8.32ns 0.32 0 0 2.02 KB
Benchmarks.Trace.SpanBenchmark - Slower :warning: Same allocations :heavy_check_mark:

Slower :warning: in #5157

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1 1.147 784.46 899.64
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0 1.124 537.60 604.40

Faster :tada: in #5157

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 1.125 543.83 483.62

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 544ns 0.236ns 0.884ns 0.00792 0 0 568 B
master StartFinishSpan netcoreapp3.1 752ns 0.662ns 2.57ns 0.00759 0 0 568 B
master StartFinishSpan net472 830ns 0.624ns 2.42ns 0.0905 0 0 570 B
master StartFinishScope net6.0 538ns 0.347ns 1.3ns 0.00967 0 0 688 B
master StartFinishScope netcoreapp3.1 785ns 0.973ns 3.37ns 0.00947 0 0 688 B
master StartFinishScope net472 1.01μs 0.868ns 3.36ns 0.103 0 0 650 B
#5157 StartFinishSpan net6.0 484ns 0.38ns 1.47ns 0.00801 0 0 568 B
#5157 StartFinishSpan netcoreapp3.1 704ns 0.619ns 2.31ns 0.00785 0 0 568 B
#5157 StartFinishSpan net472 789ns 0.805ns 2.9ns 0.0905 0 0 570 B
#5157 StartFinishScope net6.0 605ns 0.469ns 1.82ns 0.0097 0 0 688 B
#5157 StartFinishScope netcoreapp3.1 899ns 1.02ns 3.97ns 0.00939 0 0 688 B
#5157 StartFinishScope net472 964ns 0.849ns 3.06ns 0.103 0 0 650 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 700ns 0.372ns 1.44ns 0.00975 0 0 688 B
master RunOnMethodBegin netcoreapp3.1 1.02μs 4.38ns 17ns 0.00916 0 0 688 B
master RunOnMethodBegin net472 1.11μs 0.989ns 3.83ns 0.103 0 0 650 B
#5157 RunOnMethodBegin net6.0 740ns 1.08ns 4.17ns 0.00965 0 0 688 B
#5157 RunOnMethodBegin netcoreapp3.1 998ns 1.15ns 4.29ns 0.00938 0 0 688 B
#5157 RunOnMethodBegin net472 1.15μs 0.546ns 2.04ns 0.103 0 0 650 B

andrewlock avatar Apr 17 '24 10:04 andrewlock

Benchmarks Report for appsec :snail:

Benchmarks for #5157 compared to master:

  • 1 benchmarks are slower, with geometric mean 1.175
  • 1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Slower :warning: Same allocations :heavy_check_mark:

Slower :warning: in #5157

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody‑netcoreapp3.1 1.175 3,908.84 4,594.36

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 72.1μs 96ns 372ns 0.0717 0 0 5.99 KB
master AllCycleSimpleBody netcoreapp3.1 63μs 82.4ns 319ns 0.0944 0 0 6.93 KB
master AllCycleSimpleBody net472 48.7μs 103ns 399ns 1.31 0 0 8.32 KB
master AllCycleMoreComplexBody net6.0 78.1μs 132ns 513ns 0.117 0 0 9.5 KB
master AllCycleMoreComplexBody netcoreapp3.1 69.3μs 125ns 485ns 0.14 0 0 10.35 KB
master AllCycleMoreComplexBody net472 55.4μs 78.1ns 303ns 1.87 0.0275 0 11.83 KB
master ObjectExtractorSimpleBody net6.0 147ns 0.134ns 0.52ns 0.00395 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 207ns 0.0971ns 0.35ns 0.00375 0 0 272 B
master ObjectExtractorSimpleBody net472 175ns 0.0957ns 0.371ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 3.05μs 2.68ns 10.4ns 0.0532 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 3.91μs 2.57ns 9.96ns 0.0507 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 3.81μs 2.18ns 8.15ns 0.603 0.00574 0 3.8 KB
#5157 AllCycleSimpleBody net6.0 72.1μs 87.1ns 337ns 0.0719 0 0 5.99 KB
#5157 AllCycleSimpleBody netcoreapp3.1 61.3μs 80.9ns 303ns 0.0916 0 0 6.93 KB
#5157 AllCycleSimpleBody net472 47.7μs 49.6ns 192ns 1.31 0 0 8.32 KB
#5157 AllCycleMoreComplexBody net6.0 79.3μs 70.5ns 264ns 0.119 0 0 9.49 KB
#5157 AllCycleMoreComplexBody netcoreapp3.1 68.8μs 106ns 412ns 0.136 0 0 10.35 KB
#5157 AllCycleMoreComplexBody net472 54.9μs 74ns 287ns 1.88 0.0272 0 11.83 KB
#5157 ObjectExtractorSimpleBody net6.0 149ns 0.108ns 0.403ns 0.00392 0 0 280 B
#5157 ObjectExtractorSimpleBody netcoreapp3.1 198ns 0.453ns 1.63ns 0.00372 0 0 272 B
#5157 ObjectExtractorSimpleBody net472 169ns 0.109ns 0.423ns 0.0446 0 0 281 B
#5157 ObjectExtractorMoreComplexBody net6.0 2.94μs 1.16ns 4.35ns 0.0527 0 0 3.78 KB
#5157 ObjectExtractorMoreComplexBody netcoreapp3.1 4.6μs 2.23ns 8.62ns 0.0505 0 0 3.69 KB
#5157 ObjectExtractorMoreComplexBody net472 3.75μs 2.05ns 7.93ns 0.602 0.00574 0 3.8 KB
Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EncodeArgs net6.0 40.7μs 25ns 96.7ns 0.451 0 0 32.4 KB
master EncodeArgs netcoreapp3.1 56.3μs 20.9ns 81.1ns 0.448 0 0 32.4 KB
master EncodeArgs net472 68.9μs 30ns 112ns 5.16 0.0688 0 32.5 KB
master EncodeLegacyArgs net6.0 74.9μs 50.4ns 174ns 0 0 0 2.14 KB
master EncodeLegacyArgs netcoreapp3.1 104μs 149ns 577ns 0 0 0 2.14 KB
master EncodeLegacyArgs net472 153μs 109ns 420ns 0.304 0 0 2.15 KB
#5157 EncodeArgs net6.0 38.8μs 17.2ns 66.6ns 0.446 0 0 32.4 KB
#5157 EncodeArgs netcoreapp3.1 55.4μs 28.6ns 107ns 0.44 0 0 32.4 KB
#5157 EncodeArgs net472 68μs 39ns 151ns 5.14 0.0681 0 32.5 KB
#5157 EncodeLegacyArgs net6.0 75.6μs 234ns 906ns 0.038 0 0 2.14 KB
#5157 EncodeLegacyArgs netcoreapp3.1 104μs 80.3ns 311ns 0 0 0 2.15 KB
#5157 EncodeLegacyArgs net472 154μs 54.4ns 211ns 0.307 0 0 2.15 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWafRealisticBenchmark net6.0 180μs 277ns 1.07μs 0 0 0 2.13 KB
master RunWafRealisticBenchmark netcoreapp3.1 194μs 150ns 580ns 0 0 0 2.1 KB
master RunWafRealisticBenchmark net472 210μs 113ns 438ns 0.316 0 0 2.17 KB
master RunWafRealisticBenchmarkWithAttack net6.0 120μs 173ns 669ns 0 0 0 1.37 KB
master RunWafRealisticBenchmarkWithAttack netcoreapp3.1 127μs 85.9ns 321ns 0 0 0 1.36 KB
master RunWafRealisticBenchmarkWithAttack net472 139μs 32.9ns 123ns 0.207 0 0 1.39 KB
#5157 RunWafRealisticBenchmark net6.0 180μs 116ns 451ns 0 0 0 2.13 KB
#5157 RunWafRealisticBenchmark netcoreapp3.1 194μs 103ns 400ns 0 0 0 2.11 KB
#5157 RunWafRealisticBenchmark net472 212μs 73.3ns 284ns 0.318 0 0 2.17 KB
#5157 RunWafRealisticBenchmarkWithAttack net6.0 120μs 52.9ns 198ns 0 0 0 1.37 KB
#5157 RunWafRealisticBenchmarkWithAttack netcoreapp3.1 129μs 74.6ns 289ns 0 0 0 1.36 KB
#5157 RunWafRealisticBenchmarkWithAttack net472 139μs 40.1ns 155ns 0.209 0 0 1.39 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed :heavy_check_mark: More allocations :warning:

More allocations :warning: in #5157

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472 59.26 KB 62.16 KB 2.9 KB 4.89%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark net6.0 60.4μs 726ns 6.89μs 0 0 0 43.44 KB
master StringConcatBenchmark netcoreapp3.1 52.7μs 255ns 987ns 0 0 0 42.64 KB
master StringConcatBenchmark net472 37.9μs 69.9ns 261ns 0 0 0 59.26 KB
master StringConcatAspectBenchmark net6.0 295μs 1.6μs 8.46μs 0 0 0 204.16 KB
master StringConcatAspectBenchmark netcoreapp3.1 311μs 1.31μs 8.36μs 0 0 0 203.71 KB
master StringConcatAspectBenchmark net472 269μs 5.87μs 57.2μs 0 0 0 221.18 KB
#5157 StringConcatBenchmark net6.0 62.5μs 684ns 6.81μs 0 0 0 43.44 KB
#5157 StringConcatBenchmark netcoreapp3.1 61μs 699ns 6.92μs 0 0 0 42.64 KB
#5157 StringConcatBenchmark net472 37.7μs 58.1ns 217ns 0 0 0 62.16 KB
#5157 StringConcatAspectBenchmark net6.0 296μs 1.63μs 9.9μs 0 0 0 204.69 KB
#5157 StringConcatAspectBenchmark netcoreapp3.1 306μs 1.45μs 5.96μs 0 0 0 203.34 KB
#5157 StringConcatAspectBenchmark net472 264μs 6.3μs 60.4μs 0 0 0 221.18 KB

andrewlock avatar Apr 17 '24 11:04 andrewlock