dd-trace-dotnet icon indicating copy to clipboard operation
dd-trace-dotnet copied to clipboard
verified publisher icon DataDog

[Profiler] Allow enabling the profiler on ARM64

Open gleocadie opened this issue 2 years ago • 29 comments

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

gleocadie avatar Feb 20 '23 10:02 gleocadie

Datadog Report

Branch report: gleocadie/activate-integration-tests-for-arm64 Commit report: 7eccb0e

:x: dd-trace-dotnet: 21 Failed (0 Known Flaky), 0 New Flaky, 234017 Passed, 786 Skipped, 22m 41.79s Wall Time

:x: Failed Tests (21)

This report shows up to 5 failed tests.

  • UseTracerServiceName - Datadog.Profiler.IntegrationTests.ApplicationInfoTest - Details

    Expand for error 
    xpected infos.Last() to be equal to (BuggyBitsService, BuggyBitsEnv, BuggyBitsVersion), but found (dotnet, Unspecified-Environment, Unspecified-Version).

  • UseTracerServiceName - Datadog.Profiler.IntegrationTests.ApplicationInfoTest - Details

    Expand for error 
    xpected infos.Last() to be equal to (BuggyBitsService, BuggyBitsEnv, BuggyBitsVersion), but found (dotnet, Unspecified-Environment, Unspecified-Version).

  • UseTracerServiceName - Datadog.Profiler.IntegrationTests.ApplicationInfoTest - Details

    Expand for error 
    xpected infos.Last() to be equal to (BuggyBitsService, BuggyBitsEnv, BuggyBitsVersion), but found (dotnet, Unspecified-Environment, Unspecified-Version).

  • CheckEndpointsAreAttached - Datadog.Profiler.IntegrationTests.CodeHotspot.CodeHotspotTest - Details

    Expand for error 
    xpected endpoints.Distinct() to be a collection with 1 item(s), but found an empty collection.

ith configuration:
 Use declared types and members
 Compare enums by value
 Include all non-private properties
 Include all non-private fields
 Match member by name (or throw)
 Without automatic conversion.
 Without automatic conversion.
..

  • CheckEndpointsAreAttached - Datadog.Profiler.IntegrationTests.CodeHotspot.CodeHotspotTest - Details

    Expand for error 
    xpected endpoints.Distinct() to be a collection with 1 item(s), but found an empty collection.

ith configuration:
 Use declared types and members
 Compare enums by value
 Include all non-private properties
 Include all non-private fields
 Match member by name (or throw)
 Without automatic conversion.
 Without automatic conversion.
..

datadog-ddstaging[bot] avatar Feb 20 '23 10:02 datadog-ddstaging[bot]

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,304ms)  : 3246, 3363
     .   : milestone, 3304,
    master - mean (3,316ms)  : 3212, 3420
     .   : milestone, 3316,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,311ms)  : 3233, 3389
     .   : milestone, 3311,
    master - mean (3,322ms)  : 3221, 3423
     .   : milestone, 3322,


gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,176ms)  : 3102, 3250
     .   : milestone, 3176,
    master - mean (3,201ms)  : 3122, 3280
     .   : milestone, 3201,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,194ms)  : 3095, 3293
     .   : milestone, 3194,
    master - mean (3,194ms)  : 3122, 3266
     .   : milestone, 3194,


gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,169ms)  : 3087, 3251
     .   : milestone, 3169,
    master - mean (3,182ms)  : 3090, 3274
     .   : milestone, 3182,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,162ms)  : 3097, 3226
     .   : milestone, 3162,
    master - mean (3,172ms)  : 3103, 3242
     .   : milestone, 3172,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (199ms)  : 185, 214
     .   : milestone, 199,
    master - mean (204ms)  : 191, 217
     .   : milestone, 204,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (200ms)  : 190, 210
     .   : milestone, 200,
    master - mean (200ms)  : 189, 211
     .   : milestone, 200,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (264ms)  : 249, 278
     .   : milestone, 264,
    master - mean (266ms)  : 250, 282
     .   : milestone, 266,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (268ms)  : 248, 288
     .   : milestone, 268,
    master - mean (267ms)  : 252, 281
     .   : milestone, 267,


gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (254ms)  : 235, 274
     .   : milestone, 254,
    master - mean (251ms)  : 238, 265
     .   : milestone, 251,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (252ms)  : 233, 271
     .   : milestone, 252,
    master - mean (253ms)  : 238, 268
     .   : milestone, 253,

andrewlock avatar Feb 20 '23 12:02 andrewlock

Benchmarks Report :snail:

Benchmarks for #3816 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.143
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces netcoreapp3.1 551μs 95.1ns 343ns 0 0 0 2.63 KB
master WriteAndFlushEnrichedTraces net472 756μs 444ns 1.72μs 0.374 0 0 3.22 KB
#3816 WriteAndFlushEnrichedTraces netcoreapp3.1 548μs 89ns 321ns 0 0 0 2.63 KB
#3816 WriteAndFlushEnrichedTraces net472 754μs 172ns 664ns 0.377 0 0 3.22 KB
Benchmarks.Trace.AppSecBodyBenchmark - Faster :tada: Same allocations :heavy_check_mark:

Faster :tada: in #3816

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.AppSecBodyBenchmark.AllCycleSimpleBody‑net472 1.143 104,334.26 91,295.89

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody netcoreapp3.1 97.7μs 550ns 3.81μs 0 0 0 1.66 KB
master AllCycleSimpleBody net472 104μs 48.5ns 188ns 0.28 0 0 1.79 KB
master AllCycleMoreComplexBody netcoreapp3.1 290μs 268ns 1.04μs 0.146 0 0 12.11 KB
master AllCycleMoreComplexBody net472 291μs 144ns 537ns 1.93 0 0 13.04 KB
master ObjectExtractorSimpleBody netcoreapp3.1 229ns 0.0919ns 0.356ns 0.00369 0 0 272 B
master ObjectExtractorSimpleBody net472 272ns 0.228ns 0.882ns 0.0574 0 0 361 B
master ObjectExtractorMoreComplexBody netcoreapp3.1 12.7μs 3.86ns 14.5ns 0.0893 0 0 6.75 KB
master ObjectExtractorMoreComplexBody net472 17.1μs 15.3ns 59.3ns 1.21 0.017 0 7.62 KB
#3816 AllCycleSimpleBody netcoreapp3.1 98.3μs 348ns 1.35μs 0 0 0 1.66 KB
#3816 AllCycleSimpleBody net472 91.1μs 169ns 632ns 0.238 0 0 1.79 KB
#3816 AllCycleMoreComplexBody netcoreapp3.1 286μs 974ns 7.61μs 0.137 0 0 12.11 KB
#3816 AllCycleMoreComplexBody net472 288μs 67.9ns 245ns 2.02 0 0 13.04 KB
#3816 ObjectExtractorSimpleBody netcoreapp3.1 228ns 0.16ns 0.598ns 0.00381 0 0 272 B
#3816 ObjectExtractorSimpleBody net472 289ns 0.161ns 0.603ns 0.0573 0 0 361 B
#3816 ObjectExtractorMoreComplexBody netcoreapp3.1 12.5μs 10.3ns 38.5ns 0.0882 0 0 6.75 KB
#3816 ObjectExtractorMoreComplexBody net472 17.3μs 28.7ns 104ns 1.2 0.0173 0 7.62 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest netcoreapp3.1 182μs 195ns 731ns 0.27 0 0 20.9 KB
master SendRequest net472 0.0399ns 0.000685ns 0.00265ns 0 0 0 0 b
#3816 SendRequest netcoreapp3.1 183μs 186ns 721ns 0.183 0 0 20.9 KB
#3816 SendRequest net472 0.0383ns 0.000514ns 0.00192ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces netcoreapp3.1 587μs 918ns 3.55μs 0.59 0 0 41.82 KB
master WriteAndFlushEnrichedTraces net472 769μs 1.79μs 6.94μs 8.28 2.64 0.377 53.26 KB
#3816 WriteAndFlushEnrichedTraces netcoreapp3.1 586μs 1.51μs 5.86μs 0.296 0 0 41.77 KB
#3816 WriteAndFlushEnrichedTraces net472 780μs 3.07μs 11.5μs 8.15 2.33 0.388 53.22 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery netcoreapp3.1 1.26μs 0.553ns 2.07ns 0.0126 0 0 912 B
master ExecuteNonQuery net472 1.61μs 0.715ns 2.67ns 0.145 0 0 915 B
#3816 ExecuteNonQuery netcoreapp3.1 1.29μs 0.762ns 2.95ns 0.0123 0 0 912 B
#3816 ExecuteNonQuery net472 1.58μs 0.777ns 2.91ns 0.145 0.000781 0 915 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch netcoreapp3.1 1.3μs 0.492ns 1.84ns 0.0143 0 0 1.07 KB
master CallElasticsearch net472 2.29μs 0.943ns 3.65ns 0.178 0 0 1.12 KB
master CallElasticsearchAsync netcoreapp3.1 1.36μs 0.756ns 2.83ns 0.0158 0 0 1.19 KB
master CallElasticsearchAsync net472 2.28μs 1.65ns 6.41ns 0.199 0.00113 0 1.26 KB
#3816 CallElasticsearch netcoreapp3.1 1.3μs 0.621ns 2.24ns 0.0149 0 0 1.07 KB
#3816 CallElasticsearch net472 2.17μs 0.967ns 3.49ns 0.179 0 0 1.12 KB
#3816 CallElasticsearchAsync netcoreapp3.1 1.3μs 0.713ns 2.76ns 0.0163 0 0 1.19 KB
#3816 CallElasticsearchAsync net472 2.43μs 1.22ns 4.71ns 0.2 0.00121 0 1.26 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync netcoreapp3.1 1.51μs 0.709ns 2.65ns 0.0181 0 0 1.32 KB
master ExecuteAsync net472 2.49μs 1.07ns 4.01ns 0.22 0.00125 0 1.39 KB
#3816 ExecuteAsync netcoreapp3.1 1.48μs 0.483ns 1.87ns 0.0182 0 0 1.32 KB
#3816 ExecuteAsync net472 2.44μs 0.971ns 3.63ns 0.219 0.00122 0 1.39 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync netcoreapp3.1 4.51μs 1.33ns 5.16ns 0.0431 0 0 3.1 KB
master SendAsync net472 7.04μs 25.5ns 98.9ns 0.541 0 0 3.4 KB
#3816 SendAsync netcoreapp3.1 4.33μs 2.35ns 9.11ns 0.0412 0 0 3.1 KB
#3816 SendAsync net472 6.85μs 2.77ns 10.7ns 0.54 0 0 3.4 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 1.96μs 0.933ns 3.61ns 0.0247 0 0 1.85 KB
master EnrichedLog net472 2.48μs 1.18ns 4.42ns 0.286 0 0 1.81 KB
#3816 EnrichedLog netcoreapp3.1 1.91μs 0.664ns 2.39ns 0.0247 0 0 1.85 KB
#3816 EnrichedLog net472 2.43μs 0.634ns 2.46ns 0.287 0 0 1.81 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 121μs 102ns 396ns 0 0 0 4.43 KB
master EnrichedLog net472 149μs 91.1ns 353ns 0.672 0.224 0 4.65 KB
#3816 EnrichedLog netcoreapp3.1 119μs 233ns 902ns 0.0595 0 0 4.43 KB
#3816 EnrichedLog net472 150μs 148ns 575ns 0.673 0.224 0 4.65 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 3.86μs 1.62ns 6.28ns 0.0523 0 0 3.91 KB
master EnrichedLog net472 5.14μs 3.06ns 11.4ns 0.568 0.00257 0 3.58 KB
#3816 EnrichedLog netcoreapp3.1 3.83μs 2.51ns 9.39ns 0.0534 0 0 3.91 KB
#3816 EnrichedLog net472 5.12μs 2.64ns 9.86ns 0.569 0.00256 0 3.58 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive netcoreapp3.1 1.58μs 2.95ns 11ns 0.0171 0 0 1.3 KB
master SendReceive net472 1.85μs 1.49ns 5.39ns 0.212 0 0 1.34 KB
#3816 SendReceive netcoreapp3.1 1.58μs 0.662ns 2.56ns 0.0173 0 0 1.3 KB
#3816 SendReceive net472 1.92μs 1.02ns 3.82ns 0.213 0.000962 0 1.34 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 3.66μs 1.82ns 7.04ns 0.0237 0 0 1.8 KB
master EnrichedLog net472 4.25μs 1.53ns 5.94ns 0.352 0 0 2.22 KB
#3816 EnrichedLog netcoreapp3.1 3.62μs 15.4ns 59.6ns 0.0245 0 0 1.8 KB
#3816 EnrichedLog net472 4.38μs 1.79ns 6.95ns 0.351 0 0 2.22 KB
Benchmarks.Trace.SpanBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan netcoreapp3.1 684ns 0.274ns 1.06ns 0.00991 0 0 736 B
master StartFinishSpan net472 988ns 0.368ns 1.42ns 0.124 0 0 778 B
master StartFinishScope netcoreapp3.1 883ns 1.03ns 3.87ns 0.0115 0 0 856 B
master StartFinishScope net472 1.17μs 0.45ns 1.74ns 0.136 0 0 859 B
#3816 StartFinishSpan netcoreapp3.1 680ns 0.193ns 0.748ns 0.0102 0 0 736 B
#3816 StartFinishSpan net472 903ns 0.188ns 0.677ns 0.123 0 0 778 B
#3816 StartFinishScope netcoreapp3.1 909ns 0.729ns 2.82ns 0.0114 0 0 856 B
#3816 StartFinishScope net472 1.12μs 0.508ns 1.9ns 0.136 0 0 859 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin netcoreapp3.1 909ns 0.405ns 1.52ns 0.0115 0 0 856 B
master RunOnMethodBegin net472 1.26μs 1.01ns 3.93ns 0.136 0 0 859 B
#3816 RunOnMethodBegin netcoreapp3.1 985ns 0.339ns 1.31ns 0.0113 0 0 856 B
#3816 RunOnMethodBegin net472 1.15μs 0.448ns 1.74ns 0.136 0 0 859 B

andrewlock avatar Feb 20 '23 12:02 andrewlock

Benchmarks Report :snail:

Benchmarks for #3816 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.128
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces netcoreapp3.1 551μs 95.1ns 343ns 0 0 0 2.63 KB
master WriteAndFlushEnrichedTraces net472 756μs 444ns 1.72μs 0.374 0 0 3.22 KB
#3816 WriteAndFlushEnrichedTraces netcoreapp3.1 570μs 1.97μs 14.2μs 0 0 0 2.62 KB
#3816 WriteAndFlushEnrichedTraces net472 755μs 278ns 1.04μs 0.374 0 0 3.22 KB
Benchmarks.Trace.AppSecBodyBenchmark - Faster :tada: Same allocations :heavy_check_mark:

Faster :tada: in #3816

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.AppSecBodyBenchmark.AllCycleSimpleBody‑net472 1.128 104,334.26 92,460.05

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody netcoreapp3.1 97.7μs 550ns 3.81μs 0 0 0 1.66 KB
master AllCycleSimpleBody net472 104μs 48.5ns 188ns 0.28 0 0 1.79 KB
master AllCycleMoreComplexBody netcoreapp3.1 290μs 268ns 1.04μs 0.146 0 0 12.11 KB
master AllCycleMoreComplexBody net472 291μs 144ns 537ns 1.93 0 0 13.04 KB
master ObjectExtractorSimpleBody netcoreapp3.1 229ns 0.0919ns 0.356ns 0.00369 0 0 272 B
master ObjectExtractorSimpleBody net472 272ns 0.228ns 0.882ns 0.0574 0 0 361 B
master ObjectExtractorMoreComplexBody netcoreapp3.1 12.7μs 3.86ns 14.5ns 0.0893 0 0 6.75 KB
master ObjectExtractorMoreComplexBody net472 17.1μs 15.3ns 59.3ns 1.21 0.017 0 7.62 KB
#3816 AllCycleSimpleBody netcoreapp3.1 98.4μs 354ns 1.28μs 0 0 0 1.66 KB
#3816 AllCycleSimpleBody net472 92.5μs 30.2ns 113ns 0.283 0 0 1.79 KB
#3816 AllCycleMoreComplexBody netcoreapp3.1 278μs 1.47μs 7.91μs 0.136 0 0 12.11 KB
#3816 AllCycleMoreComplexBody net472 306μs 424ns 1.64μs 2.02 0 0 13.04 KB
#3816 ObjectExtractorSimpleBody netcoreapp3.1 237ns 0.117ns 0.453ns 0.0037 0 0 272 B
#3816 ObjectExtractorSimpleBody net472 272ns 0.138ns 0.517ns 0.0574 0 0 361 B
#3816 ObjectExtractorMoreComplexBody netcoreapp3.1 13μs 5.37ns 20.1ns 0.0912 0 0 6.75 KB
#3816 ObjectExtractorMoreComplexBody net472 16.3μs 10.4ns 39ns 1.21 0.0162 0 7.62 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest netcoreapp3.1 182μs 195ns 731ns 0.27 0 0 20.9 KB
master SendRequest net472 0.0399ns 0.000685ns 0.00265ns 0 0 0 0 b
#3816 SendRequest netcoreapp3.1 181μs 181ns 676ns 0.272 0 0 20.9 KB
#3816 SendRequest net472 0.0397ns 0.000346ns 0.00129ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces netcoreapp3.1 587μs 918ns 3.55μs 0.59 0 0 41.82 KB
master WriteAndFlushEnrichedTraces net472 769μs 1.79μs 6.94μs 8.28 2.64 0.377 53.26 KB
#3816 WriteAndFlushEnrichedTraces netcoreapp3.1 601μs 652ns 2.44μs 0.3 0 0 41.91 KB
#3816 WriteAndFlushEnrichedTraces net472 785μs 3.42μs 13.3μs 8.23 2.62 0.374 53.26 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery netcoreapp3.1 1.26μs 0.553ns 2.07ns 0.0126 0 0 912 B
master ExecuteNonQuery net472 1.61μs 0.715ns 2.67ns 0.145 0 0 915 B
#3816 ExecuteNonQuery netcoreapp3.1 1.25μs 0.321ns 1.24ns 0.012 0 0 912 B
#3816 ExecuteNonQuery net472 1.6μs 0.245ns 0.95ns 0.145 0 0 915 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch netcoreapp3.1 1.3μs 0.492ns 1.84ns 0.0143 0 0 1.07 KB
master CallElasticsearch net472 2.29μs 0.943ns 3.65ns 0.178 0 0 1.12 KB
master CallElasticsearchAsync netcoreapp3.1 1.36μs 0.756ns 2.83ns 0.0158 0 0 1.19 KB
master CallElasticsearchAsync net472 2.28μs 1.65ns 6.41ns 0.199 0.00113 0 1.26 KB
#3816 CallElasticsearch netcoreapp3.1 1.23μs 0.655ns 2.45ns 0.0148 0 0 1.07 KB
#3816 CallElasticsearch net472 2.33μs 11ns 41.1ns 0.178 0 0 1.12 KB
#3816 CallElasticsearchAsync netcoreapp3.1 1.4μs 1.06ns 3.83ns 0.0161 0 0 1.19 KB
#3816 CallElasticsearchAsync net472 2.46μs 6.42ns 37.5ns 0.199 0 0 1.26 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync netcoreapp3.1 1.51μs 0.709ns 2.65ns 0.0181 0 0 1.32 KB
master ExecuteAsync net472 2.49μs 1.07ns 4.01ns 0.22 0.00125 0 1.39 KB
#3816 ExecuteAsync netcoreapp3.1 1.6μs 12ns 113ns 0.0179 0 0 1.32 KB
#3816 ExecuteAsync net472 2.45μs 1.31ns 5.07ns 0.22 0.00122 0 1.39 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync netcoreapp3.1 4.51μs 1.33ns 5.16ns 0.0431 0 0 3.1 KB
master SendAsync net472 7.04μs 25.5ns 98.9ns 0.541 0 0 3.4 KB
#3816 SendAsync netcoreapp3.1 4.38μs 1.99ns 7.43ns 0.0416 0 0 3.1 KB
#3816 SendAsync net472 6.93μs 2.87ns 11.1ns 0.539 0 0 3.4 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 1.96μs 0.933ns 3.61ns 0.0247 0 0 1.85 KB
master EnrichedLog net472 2.48μs 1.18ns 4.42ns 0.286 0 0 1.81 KB
#3816 EnrichedLog netcoreapp3.1 1.91μs 0.919ns 3.44ns 0.025 0 0 1.85 KB
#3816 EnrichedLog net472 2.46μs 1.52ns 5.88ns 0.287 0 0 1.81 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 121μs 102ns 396ns 0 0 0 4.43 KB
master EnrichedLog net472 149μs 91.1ns 353ns 0.672 0.224 0 4.65 KB
#3816 EnrichedLog netcoreapp3.1 121μs 151ns 584ns 0 0 0 4.43 KB
#3816 EnrichedLog net472 150μs 75ns 290ns 0.682 0.227 0 4.65 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 3.86μs 1.62ns 6.28ns 0.0523 0 0 3.91 KB
master EnrichedLog net472 5.14μs 3.06ns 11.4ns 0.568 0.00257 0 3.58 KB
#3816 EnrichedLog netcoreapp3.1 3.94μs 1.56ns 6.03ns 0.0538 0 0 3.91 KB
#3816 EnrichedLog net472 5.09μs 1.82ns 7.07ns 0.567 0.00254 0 3.58 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive netcoreapp3.1 1.58μs 2.95ns 11ns 0.0171 0 0 1.3 KB
master SendReceive net472 1.85μs 1.49ns 5.39ns 0.212 0 0 1.34 KB
#3816 SendReceive netcoreapp3.1 1.51μs 0.831ns 3.11ns 0.0174 0 0 1.3 KB
#3816 SendReceive net472 1.88μs 0.606ns 2.1ns 0.213 0 0 1.34 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 3.66μs 1.82ns 7.04ns 0.0237 0 0 1.8 KB
master EnrichedLog net472 4.25μs 1.53ns 5.94ns 0.352 0 0 2.22 KB
#3816 EnrichedLog netcoreapp3.1 3.61μs 1.22ns 4.55ns 0.0233 0 0 1.8 KB
#3816 EnrichedLog net472 4.4μs 1.4ns 5.04ns 0.352 0 0 2.22 KB
Benchmarks.Trace.SpanBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan netcoreapp3.1 684ns 0.274ns 1.06ns 0.00991 0 0 736 B
master StartFinishSpan net472 988ns 0.368ns 1.42ns 0.124 0 0 778 B
master StartFinishScope netcoreapp3.1 883ns 1.03ns 3.87ns 0.0115 0 0 856 B
master StartFinishScope net472 1.17μs 0.45ns 1.74ns 0.136 0 0 859 B
#3816 StartFinishSpan netcoreapp3.1 685ns 1.06ns 3.97ns 0.0102 0 0 736 B
#3816 StartFinishSpan net472 918ns 0.349ns 1.35ns 0.124 0 0 778 B
#3816 StartFinishScope netcoreapp3.1 856ns 0.758ns 2.83ns 0.0116 0 0 856 B
#3816 StartFinishScope net472 1.11μs 0.5ns 1.94ns 0.136 0 0 859 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin netcoreapp3.1 909ns 0.405ns 1.52ns 0.0115 0 0 856 B
master RunOnMethodBegin net472 1.26μs 1.01ns 3.93ns 0.136 0 0 859 B
#3816 RunOnMethodBegin netcoreapp3.1 946ns 0.348ns 1.3ns 0.0118 0 0 856 B
#3816 RunOnMethodBegin net472 1.18μs 0.295ns 1.1ns 0.136 0 0 859 B

andrewlock avatar Feb 20 '23 13:02 andrewlock

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,327ms)  : 3240, 3414
     .   : milestone, 3327,
    master - mean (3,316ms)  : 3212, 3420
     .   : milestone, 3316,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,334ms)  : 3238, 3429
     .   : milestone, 3334,
    master - mean (3,322ms)  : 3221, 3423
     .   : milestone, 3322,


gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,183ms)  : 3113, 3253
     .   : milestone, 3183,
    master - mean (3,201ms)  : 3122, 3280
     .   : milestone, 3201,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,191ms)  : 3113, 3270
     .   : milestone, 3191,
    master - mean (3,194ms)  : 3122, 3266
     .   : milestone, 3194,


gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,160ms)  : 3102, 3219
     .   : milestone, 3160,
    master - mean (3,182ms)  : 3090, 3274
     .   : milestone, 3182,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,179ms)  : 3093, 3266
     .   : milestone, 3179,
    master - mean (3,172ms)  : 3103, 3242
     .   : milestone, 3172,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (201ms)  : 192, 210
     .   : milestone, 201,
    master - mean (204ms)  : 191, 217
     .   : milestone, 204,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (199ms)  : 188, 210
     .   : milestone, 199,
    master - mean (200ms)  : 189, 211
     .   : milestone, 200,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (265ms)  : 250, 279
     .   : milestone, 265,
    master - mean (266ms)  : 250, 282
     .   : milestone, 266,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (267ms)  : 255, 280
     .   : milestone, 267,
    master - mean (267ms)  : 252, 281
     .   : milestone, 267,


gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (256ms)  : 237, 276
     .   : milestone, 256,
    master - mean (251ms)  : 238, 265
     .   : milestone, 251,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (254ms)  : 239, 270
     .   : milestone, 254,
    master - mean (253ms)  : 238, 268
     .   : milestone, 253,

andrewlock avatar Feb 20 '23 13:02 andrewlock

Benchmarks Report :snail:

Benchmarks for #3816 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.122
  • 1 benchmarks are slower, with geometric mean 1.158
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces netcoreapp3.1 551μs 95.1ns 343ns 0 0 0 2.63 KB
master WriteAndFlushEnrichedTraces net472 756μs 444ns 1.72μs 0.374 0 0 3.22 KB
#3816 WriteAndFlushEnrichedTraces netcoreapp3.1 546μs 157ns 609ns 0 0 0 2.63 KB
#3816 WriteAndFlushEnrichedTraces net472 755μs 282ns 1.09μs 0.374 0 0 3.22 KB
Benchmarks.Trace.AppSecBodyBenchmark - Slower :warning: Same allocations :heavy_check_mark:

Slower :warning: in #3816

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑netcoreapp3.1 1.158 229.21 265.43

Faster :tada: in #3816

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.AppSecBodyBenchmark.AllCycleSimpleBody‑net472 1.122 104,334.26 92,959.73

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody netcoreapp3.1 97.7μs 550ns 3.81μs 0 0 0 1.66 KB
master AllCycleSimpleBody net472 104μs 48.5ns 188ns 0.28 0 0 1.79 KB
master AllCycleMoreComplexBody netcoreapp3.1 290μs 268ns 1.04μs 0.146 0 0 12.11 KB
master AllCycleMoreComplexBody net472 291μs 144ns 537ns 1.93 0 0 13.04 KB
master ObjectExtractorSimpleBody netcoreapp3.1 229ns 0.0919ns 0.356ns 0.00369 0 0 272 B
master ObjectExtractorSimpleBody net472 272ns 0.228ns 0.882ns 0.0574 0 0 361 B
master ObjectExtractorMoreComplexBody netcoreapp3.1 12.7μs 3.86ns 14.5ns 0.0893 0 0 6.75 KB
master ObjectExtractorMoreComplexBody net472 17.1μs 15.3ns 59.3ns 1.21 0.017 0 7.62 KB
#3816 AllCycleSimpleBody netcoreapp3.1 103μs 236ns 883ns 0 0 0 1.66 KB
#3816 AllCycleSimpleBody net472 92.9μs 55.1ns 213ns 0.278 0 0 1.79 KB
#3816 AllCycleMoreComplexBody netcoreapp3.1 288μs 1.16μs 4.48μs 0.146 0 0 12.11 KB
#3816 AllCycleMoreComplexBody net472 289μs 60.9ns 211ns 2.02 0 0 13.04 KB
#3816 ObjectExtractorSimpleBody netcoreapp3.1 266ns 0.159ns 0.595ns 0.00373 0 0 272 B
#3816 ObjectExtractorSimpleBody net472 275ns 0.524ns 2.03ns 0.0574 0 0 361 B
#3816 ObjectExtractorMoreComplexBody netcoreapp3.1 13.1μs 4.53ns 17ns 0.0916 0 0 6.75 KB
#3816 ObjectExtractorMoreComplexBody net472 16.6μs 10.8ns 42ns 1.21 0.0163 0 7.62 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest netcoreapp3.1 182μs 195ns 731ns 0.27 0 0 20.9 KB
master SendRequest net472 0.0399ns 0.000685ns 0.00265ns 0 0 0 0 b
#3816 SendRequest netcoreapp3.1 181μs 179ns 694ns 0.182 0 0 20.9 KB
#3816 SendRequest net472 0.0373ns 0.000488ns 0.00189ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces netcoreapp3.1 587μs 918ns 3.55μs 0.59 0 0 41.82 KB
master WriteAndFlushEnrichedTraces net472 769μs 1.79μs 6.94μs 8.28 2.64 0.377 53.26 KB
#3816 WriteAndFlushEnrichedTraces netcoreapp3.1 592μs 413ns 1.6μs 0.598 0 0 41.65 KB
#3816 WriteAndFlushEnrichedTraces net472 783μs 2.91μs 11.3μs 8.25 2.36 0.393 53.21 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery netcoreapp3.1 1.26μs 0.553ns 2.07ns 0.0126 0 0 912 B
master ExecuteNonQuery net472 1.61μs 0.715ns 2.67ns 0.145 0 0 915 B
#3816 ExecuteNonQuery netcoreapp3.1 1.24μs 0.6ns 2.24ns 0.0124 0 0 912 B
#3816 ExecuteNonQuery net472 1.67μs 0.438ns 1.52ns 0.145 0 0 915 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch netcoreapp3.1 1.3μs 0.492ns 1.84ns 0.0143 0 0 1.07 KB
master CallElasticsearch net472 2.29μs 0.943ns 3.65ns 0.178 0 0 1.12 KB
master CallElasticsearchAsync netcoreapp3.1 1.36μs 0.756ns 2.83ns 0.0158 0 0 1.19 KB
master CallElasticsearchAsync net472 2.28μs 1.65ns 6.41ns 0.199 0.00113 0 1.26 KB
#3816 CallElasticsearch netcoreapp3.1 1.26μs 0.37ns 1.38ns 0.0146 0 0 1.07 KB
#3816 CallElasticsearch net472 2.19μs 1.11ns 4.3ns 0.178 0 0 1.12 KB
#3816 CallElasticsearchAsync netcoreapp3.1 1.44μs 2.69ns 10.4ns 0.0159 0 0 1.19 KB
#3816 CallElasticsearchAsync net472 2.34μs 1.14ns 4.42ns 0.199 0.00117 0 1.26 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync netcoreapp3.1 1.51μs 0.709ns 2.65ns 0.0181 0 0 1.32 KB
master ExecuteAsync net472 2.49μs 1.07ns 4.01ns 0.22 0.00125 0 1.39 KB
#3816 ExecuteAsync netcoreapp3.1 1.46μs 0.744ns 2.78ns 0.0175 0 0 1.32 KB
#3816 ExecuteAsync net472 2.34μs 0.554ns 2.07ns 0.22 0.00116 0 1.39 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync netcoreapp3.1 4.51μs 1.33ns 5.16ns 0.0431 0 0 3.1 KB
master SendAsync net472 7.04μs 25.5ns 98.9ns 0.541 0 0 3.4 KB
#3816 SendAsync netcoreapp3.1 4.39μs 4.01ns 15ns 0.0417 0 0 3.1 KB
#3816 SendAsync net472 6.89μs 2.84ns 11ns 0.541 0 0 3.4 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 1.96μs 0.933ns 3.61ns 0.0247 0 0 1.85 KB
master EnrichedLog net472 2.48μs 1.18ns 4.42ns 0.286 0 0 1.81 KB
#3816 EnrichedLog netcoreapp3.1 1.91μs 1.1ns 4.12ns 0.0258 0 0 1.85 KB
#3816 EnrichedLog net472 2.49μs 1.92ns 7.45ns 0.287 0 0 1.81 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 121μs 102ns 396ns 0 0 0 4.43 KB
master EnrichedLog net472 149μs 91.1ns 353ns 0.672 0.224 0 4.65 KB
#3816 EnrichedLog netcoreapp3.1 119μs 175ns 654ns 0.059 0 0 4.43 KB
#3816 EnrichedLog net472 149μs 329ns 1.19μs 0.666 0.222 0 4.65 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 3.86μs 1.62ns 6.28ns 0.0523 0 0 3.91 KB
master EnrichedLog net472 5.14μs 3.06ns 11.4ns 0.568 0.00257 0 3.58 KB
#3816 EnrichedLog netcoreapp3.1 3.88μs 2.4ns 9.29ns 0.0526 0 0 3.91 KB
#3816 EnrichedLog net472 5.15μs 2.25ns 8.71ns 0.568 0.00257 0 3.58 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive netcoreapp3.1 1.58μs 2.95ns 11ns 0.0171 0 0 1.3 KB
master SendReceive net472 1.85μs 1.49ns 5.39ns 0.212 0 0 1.34 KB
#3816 SendReceive netcoreapp3.1 1.5μs 0.585ns 2.19ns 0.0174 0 0 1.3 KB
#3816 SendReceive net472 1.88μs 1.38ns 5.17ns 0.213 0.000937 0 1.34 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 3.66μs 1.82ns 7.04ns 0.0237 0 0 1.8 KB
master EnrichedLog net472 4.25μs 1.53ns 5.94ns 0.352 0 0 2.22 KB
#3816 EnrichedLog netcoreapp3.1 3.72μs 1.53ns 5.73ns 0.0242 0 0 1.8 KB
#3816 EnrichedLog net472 4.46μs 1.33ns 5.15ns 0.353 0 0 2.22 KB
Benchmarks.Trace.SpanBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan netcoreapp3.1 684ns 0.274ns 1.06ns 0.00991 0 0 736 B
master StartFinishSpan net472 988ns 0.368ns 1.42ns 0.124 0 0 778 B
master StartFinishScope netcoreapp3.1 883ns 1.03ns 3.87ns 0.0115 0 0 856 B
master StartFinishScope net472 1.17μs 0.45ns 1.74ns 0.136 0 0 859 B
#3816 StartFinishSpan netcoreapp3.1 692ns 0.419ns 1.62ns 0.0101 0 0 736 B
#3816 StartFinishSpan net472 974ns 0.52ns 1.87ns 0.124 0 0 778 B
#3816 StartFinishScope netcoreapp3.1 808ns 0.3ns 1.12ns 0.0116 0 0 856 B
#3816 StartFinishScope net472 1.06μs 0.197ns 0.736ns 0.136 0 0 859 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin netcoreapp3.1 909ns 0.405ns 1.52ns 0.0115 0 0 856 B
master RunOnMethodBegin net472 1.26μs 1.01ns 3.93ns 0.136 0 0 859 B
#3816 RunOnMethodBegin netcoreapp3.1 910ns 1.08ns 3.91ns 0.0119 0 0 856 B
#3816 RunOnMethodBegin net472 1.3μs 0.366ns 1.42ns 0.136 0 0 859 B

andrewlock avatar Feb 20 '23 14:02 andrewlock

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,314ms)  : 3240, 3389
     .   : milestone, 3314,
    master - mean (3,316ms)  : 3212, 3420
     .   : milestone, 3316,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,325ms)  : 3228, 3422
     .   : milestone, 3325,
    master - mean (3,322ms)  : 3221, 3423
     .   : milestone, 3322,


gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,190ms)  : 3113, 3268
     .   : milestone, 3190,
    master - mean (3,201ms)  : 3122, 3280
     .   : milestone, 3201,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,199ms)  : 3093, 3305
     .   : milestone, 3199,
    master - mean (3,194ms)  : 3122, 3266
     .   : milestone, 3194,


gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,177ms)  : 3082, 3271
     .   : milestone, 3177,
    master - mean (3,182ms)  : 3090, 3274
     .   : milestone, 3182,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,180ms)  : 3083, 3277
     .   : milestone, 3180,
    master - mean (3,172ms)  : 3103, 3242
     .   : milestone, 3172,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (202ms)  : 192, 212
     .   : milestone, 202,
    master - mean (204ms)  : 191, 217
     .   : milestone, 204,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (201ms)  : 191, 211
     .   : milestone, 201,
    master - mean (200ms)  : 189, 211
     .   : milestone, 200,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (267ms)  : 251, 283
     .   : milestone, 267,
    master - mean (266ms)  : 250, 282
     .   : milestone, 266,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (267ms)  : 257, 278
     .   : milestone, 267,
    master - mean (267ms)  : 252, 281
     .   : milestone, 267,


gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (255ms)  : 240, 269
     .   : milestone, 255,
    master - mean (251ms)  : 238, 265
     .   : milestone, 251,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (254ms)  : 242, 267
     .   : milestone, 254,
    master - mean (253ms)  : 238, 268
     .   : milestone, 253,

andrewlock avatar Feb 20 '23 14:02 andrewlock

Benchmarks Report :snail:

Benchmarks for #3816 compared to master:

  • 1 benchmarks are slower, with geometric mean 1.237
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces netcoreapp3.1 551μs 95.1ns 343ns 0 0 0 2.63 KB
master WriteAndFlushEnrichedTraces net472 756μs 444ns 1.72μs 0.374 0 0 3.22 KB
#3816 WriteAndFlushEnrichedTraces netcoreapp3.1 545μs 111ns 415ns 0 0 0 2.62 KB
#3816 WriteAndFlushEnrichedTraces net472 752μs 186ns 696ns 0.372 0 0 3.22 KB
Benchmarks.Trace.AppSecBodyBenchmark - Slower :warning: Same allocations :heavy_check_mark:

Slower :warning: in #3816

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑netcoreapp3.1 1.237 229.21 283.47

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody netcoreapp3.1 97.7μs 550ns 3.81μs 0 0 0 1.66 KB
master AllCycleSimpleBody net472 104μs 48.5ns 188ns 0.28 0 0 1.79 KB
master AllCycleMoreComplexBody netcoreapp3.1 290μs 268ns 1.04μs 0.146 0 0 12.11 KB
master AllCycleMoreComplexBody net472 291μs 144ns 537ns 1.93 0 0 13.04 KB
master ObjectExtractorSimpleBody netcoreapp3.1 229ns 0.0919ns 0.356ns 0.00369 0 0 272 B
master ObjectExtractorSimpleBody net472 272ns 0.228ns 0.882ns 0.0574 0 0 361 B
master ObjectExtractorMoreComplexBody netcoreapp3.1 12.7μs 3.86ns 14.5ns 0.0893 0 0 6.75 KB
master ObjectExtractorMoreComplexBody net472 17.1μs 15.3ns 59.3ns 1.21 0.017 0 7.62 KB
#3816 AllCycleSimpleBody netcoreapp3.1 99.7μs 106ns 411ns 0 0 0 1.66 KB
#3816 AllCycleSimpleBody net472 100μs 104ns 401ns 0.277 0 0 1.79 KB
#3816 AllCycleMoreComplexBody netcoreapp3.1 285μs 1.28μs 5.56μs 0.144 0 0 12.11 KB
#3816 AllCycleMoreComplexBody net472 288μs 93ns 348ns 2.02 0 0 13.04 KB
#3816 ObjectExtractorSimpleBody netcoreapp3.1 284ns 0.492ns 1.91ns 0.00369 0 0 272 B
#3816 ObjectExtractorSimpleBody net472 273ns 0.274ns 1.03ns 0.0573 0 0 361 B
#3816 ObjectExtractorMoreComplexBody netcoreapp3.1 12.8μs 5.81ns 21.7ns 0.0896 0 0 6.75 KB
#3816 ObjectExtractorMoreComplexBody net472 17.7μs 17.6ns 65.8ns 1.21 0.0175 0 7.62 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest netcoreapp3.1 182μs 195ns 731ns 0.27 0 0 20.9 KB
master SendRequest net472 0.0399ns 0.000685ns 0.00265ns 0 0 0 0 b
#3816 SendRequest netcoreapp3.1 181μs 249ns 933ns 0.273 0 0 20.9 KB
#3816 SendRequest net472 0.0379ns 0.000736ns 0.00275ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces netcoreapp3.1 587μs 918ns 3.55μs 0.59 0 0 41.82 KB
master WriteAndFlushEnrichedTraces net472 769μs 1.79μs 6.94μs 8.28 2.64 0.377 53.26 KB
#3816 WriteAndFlushEnrichedTraces netcoreapp3.1 582μs 627ns 2.43μs 0.579 0 0 41.73 KB
#3816 WriteAndFlushEnrichedTraces net472 753μs 3μs 11.6μs 8.46 2.57 0.368 53.26 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery netcoreapp3.1 1.26μs 0.553ns 2.07ns 0.0126 0 0 912 B
master ExecuteNonQuery net472 1.61μs 0.715ns 2.67ns 0.145 0 0 915 B
#3816 ExecuteNonQuery netcoreapp3.1 1.27μs 0.366ns 1.37ns 0.0122 0 0 912 B
#3816 ExecuteNonQuery net472 1.62μs 0.539ns 2.02ns 0.145 0 0 915 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch netcoreapp3.1 1.3μs 0.492ns 1.84ns 0.0143 0 0 1.07 KB
master CallElasticsearch net472 2.29μs 0.943ns 3.65ns 0.178 0 0 1.12 KB
master CallElasticsearchAsync netcoreapp3.1 1.36μs 0.756ns 2.83ns 0.0158 0 0 1.19 KB
master CallElasticsearchAsync net472 2.28μs 1.65ns 6.41ns 0.199 0.00113 0 1.26 KB
#3816 CallElasticsearch netcoreapp3.1 1.27μs 0.693ns 2.68ns 0.0146 0 0 1.07 KB
#3816 CallElasticsearch net472 2.32μs 2.18ns 8.44ns 0.178 0 0 1.12 KB
#3816 CallElasticsearchAsync netcoreapp3.1 1.33μs 3.99ns 14.4ns 0.016 0 0 1.19 KB
#3816 CallElasticsearchAsync net472 2.45μs 1.35ns 5.21ns 0.199 0.00122 0 1.26 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync netcoreapp3.1 1.51μs 0.709ns 2.65ns 0.0181 0 0 1.32 KB
master ExecuteAsync net472 2.49μs 1.07ns 4.01ns 0.22 0.00125 0 1.39 KB
#3816 ExecuteAsync netcoreapp3.1 1.48μs 0.3ns 1.08ns 0.0178 0 0 1.32 KB
#3816 ExecuteAsync net472 2.45μs 0.264ns 0.913ns 0.22 0.00122 0 1.39 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync netcoreapp3.1 4.51μs 1.33ns 5.16ns 0.0431 0 0 3.1 KB
master SendAsync net472 7.04μs 25.5ns 98.9ns 0.541 0 0 3.4 KB
#3816 SendAsync netcoreapp3.1 4.37μs 2.05ns 7.94ns 0.0415 0 0 3.1 KB
#3816 SendAsync net472 6.93μs 3.86ns 14.9ns 0.537 0 0 3.4 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 1.96μs 0.933ns 3.61ns 0.0247 0 0 1.85 KB
master EnrichedLog net472 2.48μs 1.18ns 4.42ns 0.286 0 0 1.81 KB
#3816 EnrichedLog netcoreapp3.1 1.92μs 1.01ns 3.89ns 0.0249 0 0 1.85 KB
#3816 EnrichedLog net472 2.48μs 0.943ns 3.53ns 0.287 0 0 1.81 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 121μs 102ns 396ns 0 0 0 4.43 KB
master EnrichedLog net472 149μs 91.1ns 353ns 0.672 0.224 0 4.65 KB
#3816 EnrichedLog netcoreapp3.1 117μs 137ns 530ns 0.0589 0 0 4.43 KB
#3816 EnrichedLog net472 151μs 141ns 548ns 0.678 0.226 0 4.65 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 3.86μs 1.62ns 6.28ns 0.0523 0 0 3.91 KB
master EnrichedLog net472 5.14μs 3.06ns 11.4ns 0.568 0.00257 0 3.58 KB
#3816 EnrichedLog netcoreapp3.1 3.8μs 2.01ns 7.24ns 0.0534 0 0 3.91 KB
#3816 EnrichedLog net472 5.15μs 2ns 7.48ns 0.568 0.00256 0 3.58 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive netcoreapp3.1 1.58μs 2.95ns 11ns 0.0171 0 0 1.3 KB
master SendReceive net472 1.85μs 1.49ns 5.39ns 0.212 0 0 1.34 KB
#3816 SendReceive netcoreapp3.1 1.56μs 0.495ns 1.85ns 0.0172 0 0 1.3 KB
#3816 SendReceive net472 1.84μs 0.833ns 3.23ns 0.213 0.000928 0 1.34 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 3.66μs 1.82ns 7.04ns 0.0237 0 0 1.8 KB
master EnrichedLog net472 4.25μs 1.53ns 5.94ns 0.352 0 0 2.22 KB
#3816 EnrichedLog netcoreapp3.1 3.75μs 1.42ns 5.3ns 0.0244 0 0 1.8 KB
#3816 EnrichedLog net472 4.38μs 1.43ns 5.36ns 0.352 0 0 2.22 KB
Benchmarks.Trace.SpanBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan netcoreapp3.1 684ns 0.274ns 1.06ns 0.00991 0 0 736 B
master StartFinishSpan net472 988ns 0.368ns 1.42ns 0.124 0 0 778 B
master StartFinishScope netcoreapp3.1 883ns 1.03ns 3.87ns 0.0115 0 0 856 B
master StartFinishScope net472 1.17μs 0.45ns 1.74ns 0.136 0 0 859 B
#3816 StartFinishSpan netcoreapp3.1 685ns 0.313ns 1.17ns 0.01 0 0 736 B
#3816 StartFinishSpan net472 904ns 0.328ns 1.27ns 0.124 0 0 778 B
#3816 StartFinishScope netcoreapp3.1 865ns 0.276ns 0.996ns 0.0117 0 0 856 B
#3816 StartFinishScope net472 1.1μs 0.267ns 1.04ns 0.136 0 0 859 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin netcoreapp3.1 909ns 0.405ns 1.52ns 0.0115 0 0 856 B
master RunOnMethodBegin net472 1.26μs 1.01ns 3.93ns 0.136 0 0 859 B
#3816 RunOnMethodBegin netcoreapp3.1 973ns 0.419ns 1.51ns 0.0116 0 0 856 B
#3816 RunOnMethodBegin net472 1.22μs 0.435ns 1.68ns 0.136 0 0 859 B

andrewlock avatar Feb 20 '23 14:02 andrewlock

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,319ms)  : 3222, 3416
     .   : milestone, 3319,
    master - mean (3,316ms)  : 3212, 3420
     .   : milestone, 3316,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,318ms)  : 3226, 3410
     .   : milestone, 3318,
    master - mean (3,322ms)  : 3221, 3423
     .   : milestone, 3322,


gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,188ms)  : 3099, 3277
     .   : milestone, 3188,
    master - mean (3,201ms)  : 3122, 3280
     .   : milestone, 3201,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,179ms)  : 3114, 3245
     .   : milestone, 3179,
    master - mean (3,194ms)  : 3122, 3266
     .   : milestone, 3194,


gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,179ms)  : 3060, 3297
     .   : milestone, 3179,
    master - mean (3,182ms)  : 3090, 3274
     .   : milestone, 3182,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,168ms)  : 3086, 3249
     .   : milestone, 3168,
    master - mean (3,172ms)  : 3103, 3242
     .   : milestone, 3172,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (213ms)  : 197, 229
     .   : milestone, 213,
    master - mean (204ms)  : 191, 217
     .   : milestone, 204,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (213ms)  : 193, 233
     .   : milestone, 213,
    master - mean (200ms)  : 189, 211
     .   : milestone, 200,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (279ms)  : 259, 299
     .   : milestone, 279,
    master - mean (266ms)  : 250, 282
     .   : milestone, 266,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (280ms)  : 260, 300
     .   : milestone, 280,
    master - mean (267ms)  : 252, 281
     .   : milestone, 267,


gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (264ms)  : 241, 287
     .   : milestone, 264,
    master - mean (251ms)  : 238, 265
     .   : milestone, 251,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (267ms)  : 241, 294
     .   : milestone, 267,
    master - mean (253ms)  : 238, 268
     .   : milestone, 253,

andrewlock avatar Feb 20 '23 15:02 andrewlock

Benchmarks Report :snail:

Benchmarks for #3816 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.148
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces netcoreapp3.1 551μs 95.1ns 343ns 0 0 0 2.63 KB
master WriteAndFlushEnrichedTraces net472 756μs 444ns 1.72μs 0.374 0 0 3.22 KB
#3816 WriteAndFlushEnrichedTraces netcoreapp3.1 560μs 232ns 870ns 0 0 0 2.62 KB
#3816 WriteAndFlushEnrichedTraces net472 750μs 137ns 529ns 0.372 0 0 3.22 KB
Benchmarks.Trace.AppSecBodyBenchmark - Faster :tada: Same allocations :heavy_check_mark:

Faster :tada: in #3816

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.AppSecBodyBenchmark.AllCycleSimpleBody‑net472 1.148 104,334.26 90,899.22

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody netcoreapp3.1 97.7μs 550ns 3.81μs 0 0 0 1.66 KB
master AllCycleSimpleBody net472 104μs 48.5ns 188ns 0.28 0 0 1.79 KB
master AllCycleMoreComplexBody netcoreapp3.1 290μs 268ns 1.04μs 0.146 0 0 12.11 KB
master AllCycleMoreComplexBody net472 291μs 144ns 537ns 1.93 0 0 13.04 KB
master ObjectExtractorSimpleBody netcoreapp3.1 229ns 0.0919ns 0.356ns 0.00369 0 0 272 B
master ObjectExtractorSimpleBody net472 272ns 0.228ns 0.882ns 0.0574 0 0 361 B
master ObjectExtractorMoreComplexBody netcoreapp3.1 12.7μs 3.86ns 14.5ns 0.0893 0 0 6.75 KB
master ObjectExtractorMoreComplexBody net472 17.1μs 15.3ns 59.3ns 1.21 0.017 0 7.62 KB
#3816 AllCycleSimpleBody netcoreapp3.1 89.2μs 293ns 1.1μs 0 0 0 1.66 KB
#3816 AllCycleSimpleBody net472 90.9μs 59.1ns 229ns 0.276 0 0 1.79 KB
#3816 AllCycleMoreComplexBody netcoreapp3.1 280μs 1.52μs 8.21μs 0.144 0 0 12.11 KB
#3816 AllCycleMoreComplexBody net472 290μs 84.6ns 305ns 2.02 0 0 13.04 KB
#3816 ObjectExtractorSimpleBody netcoreapp3.1 233ns 0.16ns 0.619ns 0.00375 0 0 272 B
#3816 ObjectExtractorSimpleBody net472 279ns 0.232ns 0.838ns 0.0573 0 0 361 B
#3816 ObjectExtractorMoreComplexBody netcoreapp3.1 12.4μs 4.25ns 15.9ns 0.0868 0 0 6.75 KB
#3816 ObjectExtractorMoreComplexBody net472 17μs 17.5ns 67.8ns 1.21 0.0169 0 7.62 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest netcoreapp3.1 182μs 195ns 731ns 0.27 0 0 20.9 KB
master SendRequest net472 0.0399ns 0.000685ns 0.00265ns 0 0 0 0 b
#3816 SendRequest netcoreapp3.1 179μs 262ns 981ns 0.266 0 0 20.9 KB
#3816 SendRequest net472 0.0377ns 0.000582ns 0.00225ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces netcoreapp3.1 587μs 918ns 3.55μs 0.59 0 0 41.82 KB
master WriteAndFlushEnrichedTraces net472 769μs 1.79μs 6.94μs 8.28 2.64 0.377 53.26 KB
#3816 WriteAndFlushEnrichedTraces netcoreapp3.1 595μs 807ns 3.13μs 0.595 0 0 41.93 KB
#3816 WriteAndFlushEnrichedTraces net472 777μs 2.01μs 7.53μs 8.2 2.34 0.391 53.32 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery netcoreapp3.1 1.26μs 0.553ns 2.07ns 0.0126 0 0 912 B
master ExecuteNonQuery net472 1.61μs 0.715ns 2.67ns 0.145 0 0 915 B
#3816 ExecuteNonQuery netcoreapp3.1 1.26μs 0.321ns 1.24ns 0.0126 0 0 912 B
#3816 ExecuteNonQuery net472 1.69μs 0.276ns 0.996ns 0.145 0 0 915 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch netcoreapp3.1 1.3μs 0.492ns 1.84ns 0.0143 0 0 1.07 KB
master CallElasticsearch net472 2.29μs 0.943ns 3.65ns 0.178 0 0 1.12 KB
master CallElasticsearchAsync netcoreapp3.1 1.36μs 0.756ns 2.83ns 0.0158 0 0 1.19 KB
master CallElasticsearchAsync net472 2.28μs 1.65ns 6.41ns 0.199 0.00113 0 1.26 KB
#3816 CallElasticsearch netcoreapp3.1 1.28μs 0.466ns 1.74ns 0.0142 0 0 1.07 KB
#3816 CallElasticsearch net472 2.1μs 0.794ns 3.07ns 0.178 0 0 1.12 KB
#3816 CallElasticsearchAsync netcoreapp3.1 1.3μs 2.83ns 11ns 0.0158 0 0 1.19 KB
#3816 CallElasticsearchAsync net472 2.32μs 2.96ns 11.5ns 0.2 0.00116 0 1.26 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync netcoreapp3.1 1.51μs 0.709ns 2.65ns 0.0181 0 0 1.32 KB
master ExecuteAsync net472 2.49μs 1.07ns 4.01ns 0.22 0.00125 0 1.39 KB
#3816 ExecuteAsync netcoreapp3.1 1.46μs 0.278ns 1ns 0.0175 0 0 1.32 KB
#3816 ExecuteAsync net472 2.41μs 0.941ns 3.65ns 0.22 0.0012 0 1.39 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync netcoreapp3.1 4.51μs 1.33ns 5.16ns 0.0431 0 0 3.1 KB
master SendAsync net472 7.04μs 25.5ns 98.9ns 0.541 0 0 3.4 KB
#3816 SendAsync netcoreapp3.1 4.35μs 2ns 7.48ns 0.0412 0 0 3.1 KB
#3816 SendAsync net472 6.91μs 3.23ns 12.1ns 0.539 0 0 3.4 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 1.96μs 0.933ns 3.61ns 0.0247 0 0 1.85 KB
master EnrichedLog net472 2.48μs 1.18ns 4.42ns 0.286 0 0 1.81 KB
#3816 EnrichedLog netcoreapp3.1 1.94μs 3.69ns 14.3ns 0.0252 0 0 1.85 KB
#3816 EnrichedLog net472 2.39μs 1.49ns 5.77ns 0.286 0 0 1.81 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 121μs 102ns 396ns 0 0 0 4.43 KB
master EnrichedLog net472 149μs 91.1ns 353ns 0.672 0.224 0 4.65 KB
#3816 EnrichedLog netcoreapp3.1 118μs 127ns 490ns 0.0586 0 0 4.43 KB
#3816 EnrichedLog net472 151μs 131ns 506ns 0.669 0.223 0 4.65 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 3.86μs 1.62ns 6.28ns 0.0523 0 0 3.91 KB
master EnrichedLog net472 5.14μs 3.06ns 11.4ns 0.568 0.00257 0 3.58 KB
#3816 EnrichedLog netcoreapp3.1 3.84μs 1.77ns 6.61ns 0.054 0 0 3.91 KB
#3816 EnrichedLog net472 5.11μs 3.42ns 12.8ns 0.567 0.00258 0 3.58 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive netcoreapp3.1 1.58μs 2.95ns 11ns 0.0171 0 0 1.3 KB
master SendReceive net472 1.85μs 1.49ns 5.39ns 0.212 0 0 1.34 KB
#3816 SendReceive netcoreapp3.1 1.51μs 0.588ns 2.28ns 0.0173 0 0 1.3 KB
#3816 SendReceive net472 1.95μs 1.55ns 6.02ns 0.212 0.000974 0 1.34 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 3.66μs 1.82ns 7.04ns 0.0237 0 0 1.8 KB
master EnrichedLog net472 4.25μs 1.53ns 5.94ns 0.352 0 0 2.22 KB
#3816 EnrichedLog netcoreapp3.1 3.62μs 2.46ns 9.19ns 0.0252 0 0 1.8 KB
#3816 EnrichedLog net472 4.37μs 1.59ns 5.93ns 0.352 0 0 2.22 KB
Benchmarks.Trace.SpanBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan netcoreapp3.1 684ns 0.274ns 1.06ns 0.00991 0 0 736 B
master StartFinishSpan net472 988ns 0.368ns 1.42ns 0.124 0 0 778 B
master StartFinishScope netcoreapp3.1 883ns 1.03ns 3.87ns 0.0115 0 0 856 B
master StartFinishScope net472 1.17μs 0.45ns 1.74ns 0.136 0 0 859 B
#3816 StartFinishSpan netcoreapp3.1 720ns 0.249ns 0.965ns 0.0101 0 0 736 B
#3816 StartFinishSpan net472 949ns 0.295ns 1.14ns 0.124 0 0 778 B
#3816 StartFinishScope netcoreapp3.1 881ns 0.383ns 1.38ns 0.0114 0 0 856 B
#3816 StartFinishScope net472 1.08μs 0.602ns 2.33ns 0.136 0 0 859 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin netcoreapp3.1 909ns 0.405ns 1.52ns 0.0115 0 0 856 B
master RunOnMethodBegin net472 1.26μs 1.01ns 3.93ns 0.136 0 0 859 B
#3816 RunOnMethodBegin netcoreapp3.1 877ns 0.363ns 1.31ns 0.012 0 0 856 B
#3816 RunOnMethodBegin net472 1.2μs 0.236ns 0.851ns 0.136 0 0 859 B

andrewlock avatar Feb 20 '23 16:02 andrewlock

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,316ms)  : 3234, 3397
     .   : milestone, 3316,
    master - mean (3,316ms)  : 3212, 3420
     .   : milestone, 3316,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,310ms)  : 3230, 3391
     .   : milestone, 3310,
    master - mean (3,322ms)  : 3221, 3423
     .   : milestone, 3322,


gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,184ms)  : 3102, 3265
     .   : milestone, 3184,
    master - mean (3,201ms)  : 3122, 3280
     .   : milestone, 3201,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,193ms)  : 3096, 3289
     .   : milestone, 3193,
    master - mean (3,194ms)  : 3122, 3266
     .   : milestone, 3194,


gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,168ms)  : 3087, 3250
     .   : milestone, 3168,
    master - mean (3,182ms)  : 3090, 3274
     .   : milestone, 3182,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,163ms)  : 3078, 3248
     .   : milestone, 3163,
    master - mean (3,172ms)  : 3103, 3242
     .   : milestone, 3172,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (209ms)  : 195, 222
     .   : milestone, 209,
    master - mean (204ms)  : 191, 217
     .   : milestone, 204,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (210ms)  : 188, 232
     .   : milestone, 210,
    master - mean (200ms)  : 189, 211
     .   : milestone, 200,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (273ms)  : 247, 299
     .   : milestone, 273,
    master - mean (266ms)  : 250, 282
     .   : milestone, 266,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (274ms)  : 260, 289
     .   : milestone, 274,
    master - mean (267ms)  : 252, 281
     .   : milestone, 267,


gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (258ms)  : 238, 278
     .   : milestone, 258,
    master - mean (251ms)  : 238, 265
     .   : milestone, 251,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (269ms)  : 249, 290
     .   : milestone, 269,
    master - mean (253ms)  : 238, 268
     .   : milestone, 253,

andrewlock avatar Feb 20 '23 16:02 andrewlock

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,317ms)  : 3237, 3398
     .   : milestone, 3317,
    master - mean (3,324ms)  : 3253, 3395
     .   : milestone, 3324,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,322ms)  : 3243, 3401
     .   : milestone, 3322,
    master - mean (3,323ms)  : 3236, 3411
     .   : milestone, 3323,


gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,182ms)  : 3097, 3268
     .   : milestone, 3182,
    master - mean (3,180ms)  : 3084, 3276
     .   : milestone, 3180,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,199ms)  : 3121, 3276
     .   : milestone, 3199,
    master - mean (3,194ms)  : 3111, 3278
     .   : milestone, 3194,


gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,175ms)  : 3098, 3252
     .   : milestone, 3175,
    master - mean (3,171ms)  : 3115, 3226
     .   : milestone, 3171,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,181ms)  : 3077, 3285
     .   : milestone, 3181,
    master - mean (3,168ms)  : 3102, 3234
     .   : milestone, 3168,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (212ms)  : 194, 230
     .   : milestone, 212,
    master - mean (208ms)  : 193, 222
     .   : milestone, 208,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (213ms)  : 187, 238
     .   : milestone, 213,
    master - mean (205ms)  : 196, 214
     .   : milestone, 205,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (272ms)  : 255, 288
     .   : milestone, 272,
    master - mean (277ms)  : 260, 295
     .   : milestone, 277,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (277ms)  : 256, 298
     .   : milestone, 277,
    master - mean (278ms)  : 256, 300
     .   : milestone, 278,


gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (262ms)  : 247, 276
     .   : milestone, 262,
    master - mean (263ms)  : 243, 282
     .   : milestone, 263,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (262ms)  : 243, 281
     .   : milestone, 262,
    master - mean (265ms)  : 247, 283
     .   : milestone, 265,

andrewlock avatar Feb 20 '23 18:02 andrewlock

Benchmarks Report :snail:

Benchmarks for #3816 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.174
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces netcoreapp3.1 549μs 579ns 2.24μs 0 0 0 2.62 KB
master WriteAndFlushEnrichedTraces net472 750μs 187ns 673ns 0.379 0 0 3.22 KB
#3816 WriteAndFlushEnrichedTraces netcoreapp3.1 555μs 266ns 1.03μs 0 0 0 2.62 KB
#3816 WriteAndFlushEnrichedTraces net472 750μs 357ns 1.34μs 0.379 0 0 3.22 KB
Benchmarks.Trace.AppSecBodyBenchmark - Faster :tada: Same allocations :heavy_check_mark:

Faster :tada: in #3816

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑netcoreapp3.1 1.174 281.98 240.25

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody netcoreapp3.1 97.9μs 566ns 4.87μs 0 0 0 1.66 KB
master AllCycleSimpleBody net472 96μs 61.9ns 240ns 0.275 0 0 1.79 KB
master AllCycleMoreComplexBody netcoreapp3.1 297μs 1.72μs 15.1μs 0.155 0 0 12.11 KB
master AllCycleMoreComplexBody net472 293μs 1.1μs 4.25μs 2.01 0 0 13.04 KB
master ObjectExtractorSimpleBody netcoreapp3.1 282ns 0.118ns 0.455ns 0.00368 0 0 272 B
master ObjectExtractorSimpleBody net472 282ns 0.594ns 2.3ns 0.0573 0 0 361 B
master ObjectExtractorMoreComplexBody netcoreapp3.1 13μs 7.03ns 25.3ns 0.0901 0 0 6.75 KB
master ObjectExtractorMoreComplexBody net472 17.8μs 14.3ns 55.3ns 1.21 0.0175 0 7.62 KB
#3816 AllCycleSimpleBody netcoreapp3.1 99.7μs 602ns 6.02μs 0 0 0 1.66 KB
#3816 AllCycleSimpleBody net472 92.8μs 23.5ns 84.8ns 0.281 0 0 1.79 KB
#3816 AllCycleMoreComplexBody netcoreapp3.1 288μs 1.35μs 5.22μs 0.138 0 0 12.11 KB
#3816 AllCycleMoreComplexBody net472 290μs 135ns 521ns 2.03 0 0 13.04 KB
#3816 ObjectExtractorSimpleBody netcoreapp3.1 240ns 0.115ns 0.43ns 0.00374 0 0 272 B
#3816 ObjectExtractorSimpleBody net472 274ns 0.121ns 0.468ns 0.0574 0 0 361 B
#3816 ObjectExtractorMoreComplexBody netcoreapp3.1 12.4μs 4.03ns 15.1ns 0.0866 0 0 6.75 KB
#3816 ObjectExtractorMoreComplexBody net472 17.1μs 12ns 45.1ns 1.2 0.0171 0 7.62 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest netcoreapp3.1 182μs 192ns 744ns 0.273 0 0 20.91 KB
master SendRequest net472 0.0389ns 0.000674ns 0.00243ns 0 0 0 0 b
#3816 SendRequest netcoreapp3.1 182μs 134ns 520ns 0.182 0 0 20.9 KB
#3816 SendRequest net472 0.0384ns 0.000481ns 0.0018ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces netcoreapp3.1 606μs 692ns 2.68μs 0.592 0 0 41.8 KB
master WriteAndFlushEnrichedTraces net472 774μs 2.54μs 9.83μs 8.33 2.65 0.379 53.23 KB
#3816 WriteAndFlushEnrichedTraces netcoreapp3.1 596μs 633ns 2.37μs 0.296 0 0 41.73 KB
#3816 WriteAndFlushEnrichedTraces net472 789μs 3.18μs 11.9μs 8.36 2.39 0.398 53.21 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery netcoreapp3.1 1.26μs 0.448ns 1.74ns 0.0125 0 0 912 B
master ExecuteNonQuery net472 1.54μs 0.43ns 1.67ns 0.145 0 0 915 B
#3816 ExecuteNonQuery netcoreapp3.1 1.26μs 0.306ns 1.1ns 0.0121 0 0 912 B
#3816 ExecuteNonQuery net472 1.6μs 0.742ns 2.77ns 0.145 0 0 915 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch netcoreapp3.1 1.27μs 0.649ns 2.34ns 0.0143 0 0 1.07 KB
master CallElasticsearch net472 2.2μs 1.21ns 4.52ns 0.178 0 0 1.12 KB
master CallElasticsearchAsync netcoreapp3.1 1.38μs 1.19ns 4.61ns 0.0159 0 0 1.19 KB
master CallElasticsearchAsync net472 2.43μs 0.47ns 1.82ns 0.2 0.0012 0 1.26 KB
#3816 CallElasticsearch netcoreapp3.1 1.26μs 0.724ns 2.8ns 0.0144 0 0 1.07 KB
#3816 CallElasticsearch net472 2.17μs 1.49ns 5.78ns 0.178 0 0 1.12 KB
#3816 CallElasticsearchAsync netcoreapp3.1 1.36μs 0.944ns 3.53ns 0.0159 0 0 1.19 KB
#3816 CallElasticsearchAsync net472 2.49μs 1.41ns 4.87ns 0.199 0.00122 0 1.26 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync netcoreapp3.1 1.47μs 0.757ns 2.83ns 0.0178 0 0 1.32 KB
master ExecuteAsync net472 2.42μs 1.03ns 3.99ns 0.219 0.00121 0 1.39 KB
#3816 ExecuteAsync netcoreapp3.1 1.48μs 0.406ns 1.57ns 0.0179 0 0 1.32 KB
#3816 ExecuteAsync net472 2.47μs 1.36ns 5.25ns 0.22 0.00123 0 1.39 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync netcoreapp3.1 4.36μs 1.56ns 5.61ns 0.0413 0 0 3.1 KB
master SendAsync net472 6.89μs 2.52ns 9.07ns 0.541 0 0 3.4 KB
#3816 SendAsync netcoreapp3.1 4.39μs 1.55ns 5.6ns 0.042 0 0 3.1 KB
#3816 SendAsync net472 7μs 2.21ns 8.57ns 0.538 0 0 3.4 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 1.94μs 0.41ns 1.59ns 0.0253 0 0 1.85 KB
master EnrichedLog net472 2.46μs 1.87ns 7.26ns 0.286 0 0 1.81 KB
#3816 EnrichedLog netcoreapp3.1 1.93μs 0.657ns 2.46ns 0.0253 0 0 1.85 KB
#3816 EnrichedLog net472 2.5μs 2.84ns 11ns 0.287 0 0 1.81 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 120μs 277ns 1.07μs 0 0 0 4.43 KB
master EnrichedLog net472 151μs 122ns 474ns 0.682 0.227 0 4.65 KB
#3816 EnrichedLog netcoreapp3.1 118μs 184ns 713ns 0.059 0 0 4.43 KB
#3816 EnrichedLog net472 148μs 82.9ns 310ns 0.734 0.22 0 4.65 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 3.93μs 1.45ns 5.24ns 0.0531 0 0 3.91 KB
master EnrichedLog net472 5.12μs 2.03ns 7.31ns 0.567 0.00255 0 3.58 KB
#3816 EnrichedLog netcoreapp3.1 3.87μs 2.01ns 7.53ns 0.0531 0 0 3.91 KB
#3816 EnrichedLog net472 5.22μs 2.39ns 9.27ns 0.566 0.0026 0 3.58 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive netcoreapp3.1 1.58μs 0.701ns 2.72ns 0.0178 0 0 1.3 KB
master SendReceive net472 1.99μs 0.622ns 2.16ns 0.213 0 0 1.34 KB
#3816 SendReceive netcoreapp3.1 1.5μs 0.515ns 1.99ns 0.0176 0 0 1.3 KB
#3816 SendReceive net472 1.9μs 2.09ns 8.08ns 0.213 0 0 1.34 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 3.51μs 1.34ns 5.17ns 0.0228 0 0 1.8 KB
master EnrichedLog net472 4.39μs 2.27ns 8.49ns 0.351 0 0 2.22 KB
#3816 EnrichedLog netcoreapp3.1 3.69μs 1.3ns 4.69ns 0.024 0 0 1.8 KB
#3816 EnrichedLog net472 4.33μs 1.99ns 7.44ns 0.352 0 0 2.22 KB
Benchmarks.Trace.SpanBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan netcoreapp3.1 688ns 0.427ns 1.54ns 0.0101 0 0 736 B
master StartFinishSpan net472 930ns 0.591ns 2.29ns 0.123 0 0 778 B
master StartFinishScope netcoreapp3.1 822ns 0.217ns 0.784ns 0.0115 0 0 856 B
master StartFinishScope net472 1.12μs 0.418ns 1.51ns 0.136 0 0 859 B
#3816 StartFinishSpan netcoreapp3.1 700ns 0.21ns 0.787ns 0.0103 0 0 736 B
#3816 StartFinishSpan net472 874ns 0.273ns 1.02ns 0.124 0 0 778 B
#3816 StartFinishScope netcoreapp3.1 857ns 0.269ns 0.969ns 0.0116 0 0 856 B
#3816 StartFinishScope net472 1.1μs 1.87ns 7.23ns 0.136 0 0 859 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin netcoreapp3.1 968ns 0.475ns 1.84ns 0.0117 0 0 856 B
master RunOnMethodBegin net472 1.25μs 0.548ns 2.05ns 0.136 0 0 859 B
#3816 RunOnMethodBegin netcoreapp3.1 941ns 0.439ns 1.7ns 0.0118 0 0 856 B
#3816 RunOnMethodBegin net472 1.2μs 1.62ns 6.26ns 0.136 0 0 859 B

andrewlock avatar Feb 20 '23 18:02 andrewlock

Code Coverage Report :bar_chart:

:heavy_check_mark: Merging #3816 into master will not change line coverage :heavy_check_mark: Merging #3816 into master will not change branch coverage :heavy_check_mark: Merging #3816 into master will not change complexity

master #3816 Change
Lines 22198 / 31829 22089 / 31829
Lines % 70% 69% 0% :heavy_check_mark:
Branches 13441 / 20889 13392 / 20889
Branches % 64% 64% 0% :heavy_check_mark:
Complexity 22502 22502 0 :heavy_check_mark:

View the full report for further details:

Datadog.Trace Breakdown :heavy_check_mark:

master #3816 Change
Lines % 70% 69% 0% :heavy_check_mark:
Branches % 64% 64% 0% :heavy_check_mark:
Complexity 22502 22502 0 :heavy_check_mark:

The following classes have significant coverage changes.

File Line coverage change Branch coverage change Complexity change
BoundLineProbeLocation -100% :no_entry: 0% :heavy_check_mark: 0 :heavy_check_mark:
Datadog.Trace.Debugger.Models.LineProbeResolveResult -100% :no_entry: 0% :heavy_check_mark: 0 :heavy_check_mark:
Datadog.Trace.Debugger.LineProbeResolver -81% :no_entry: -82% :no_entry: 0 :heavy_check_mark:
Datadog.Trace.Debugger.Configurations.Models.SnapshotSegment -80% :no_entry: -100% :no_entry: 0 :heavy_check_mark:
Datadog.Trace.Pdb.DatadogPdbReader -20% :no_entry: -38% :no_entry: 0 :heavy_check_mark:
Datadog.Trace.Ci.GitInfo -17% :no_entry: -10% :no_entry: 0 :heavy_check_mark:
Datadog.Trace.Debugger.Configurations.Models.Where -8% :no_entry: 0% :heavy_check_mark: 0 :heavy_check_mark:

View the full reports for further details:

andrewlock avatar Feb 20 '23 18:02 andrewlock

Benchmarks Report :snail:

Benchmarks for #3816 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.218
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces netcoreapp3.1 549μs 579ns 2.24μs 0 0 0 2.62 KB
master WriteAndFlushEnrichedTraces net472 750μs 187ns 673ns 0.379 0 0 3.22 KB
#3816 WriteAndFlushEnrichedTraces netcoreapp3.1 558μs 156ns 563ns 0 0 0 2.63 KB
#3816 WriteAndFlushEnrichedTraces net472 749μs 233ns 901ns 0.379 0 0 3.22 KB
Benchmarks.Trace.AppSecBodyBenchmark - Faster :tada: Same allocations :heavy_check_mark:

Faster :tada: in #3816

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑netcoreapp3.1 1.218 281.98 231.50

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody netcoreapp3.1 97.9μs 566ns 4.87μs 0 0 0 1.66 KB
master AllCycleSimpleBody net472 96μs 61.9ns 240ns 0.275 0 0 1.79 KB
master AllCycleMoreComplexBody netcoreapp3.1 297μs 1.72μs 15.1μs 0.155 0 0 12.11 KB
master AllCycleMoreComplexBody net472 293μs 1.1μs 4.25μs 2.01 0 0 13.04 KB
master ObjectExtractorSimpleBody netcoreapp3.1 282ns 0.118ns 0.455ns 0.00368 0 0 272 B
master ObjectExtractorSimpleBody net472 282ns 0.594ns 2.3ns 0.0573 0 0 361 B
master ObjectExtractorMoreComplexBody netcoreapp3.1 13μs 7.03ns 25.3ns 0.0901 0 0 6.75 KB
master ObjectExtractorMoreComplexBody net472 17.8μs 14.3ns 55.3ns 1.21 0.0175 0 7.62 KB
#3816 AllCycleSimpleBody netcoreapp3.1 101μs 369ns 1.43μs 0 0 0 1.66 KB
#3816 AllCycleSimpleBody net472 101μs 558ns 3.62μs 0.273 0 0 1.79 KB
#3816 AllCycleMoreComplexBody netcoreapp3.1 281μs 1.48μs 7.83μs 0.144 0 0 12.11 KB
#3816 AllCycleMoreComplexBody net472 296μs 1.46μs 6.36μs 2.01 0 0 13.04 KB
#3816 ObjectExtractorSimpleBody netcoreapp3.1 231ns 0.155ns 0.6ns 0.00371 0 0 272 B
#3816 ObjectExtractorSimpleBody net472 274ns 0.24ns 0.928ns 0.0573 0 0 361 B
#3816 ObjectExtractorMoreComplexBody netcoreapp3.1 12.7μs 3.17ns 11.4ns 0.0885 0 0 6.75 KB
#3816 ObjectExtractorMoreComplexBody net472 16.4μs 16.2ns 62.7ns 1.2 0.0163 0 7.62 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest netcoreapp3.1 182μs 192ns 744ns 0.273 0 0 20.91 KB
master SendRequest net472 0.0389ns 0.000674ns 0.00243ns 0 0 0 0 b
#3816 SendRequest netcoreapp3.1 182μs 189ns 733ns 0.274 0 0 20.9 KB
#3816 SendRequest net472 0.0379ns 0.000376ns 0.00141ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces netcoreapp3.1 606μs 692ns 2.68μs 0.592 0 0 41.8 KB
master WriteAndFlushEnrichedTraces net472 774μs 2.54μs 9.83μs 8.33 2.65 0.379 53.23 KB
#3816 WriteAndFlushEnrichedTraces netcoreapp3.1 587μs 660ns 2.56μs 0.581 0 0 41.68 KB
#3816 WriteAndFlushEnrichedTraces net472 778μs 2.61μs 10.1μs 8.1 2.31 0.386 53.24 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery netcoreapp3.1 1.26μs 0.448ns 1.74ns 0.0125 0 0 912 B
master ExecuteNonQuery net472 1.54μs 0.43ns 1.67ns 0.145 0 0 915 B
#3816 ExecuteNonQuery netcoreapp3.1 1.29μs 0.531ns 2.06ns 0.0123 0 0 912 B
#3816 ExecuteNonQuery net472 1.62μs 0.562ns 2.1ns 0.145 0.000812 0 915 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch netcoreapp3.1 1.27μs 0.649ns 2.34ns 0.0143 0 0 1.07 KB
master CallElasticsearch net472 2.2μs 1.21ns 4.52ns 0.178 0 0 1.12 KB
master CallElasticsearchAsync netcoreapp3.1 1.38μs 1.19ns 4.61ns 0.0159 0 0 1.19 KB
master CallElasticsearchAsync net472 2.43μs 0.47ns 1.82ns 0.2 0.0012 0 1.26 KB
#3816 CallElasticsearch netcoreapp3.1 1.31μs 0.407ns 1.52ns 0.0146 0 0 1.07 KB
#3816 CallElasticsearch net472 2.27μs 2.04ns 7.64ns 0.178 0 0 1.12 KB
#3816 CallElasticsearchAsync netcoreapp3.1 1.37μs 0.468ns 1.81ns 0.0164 0 0 1.19 KB
#3816 CallElasticsearchAsync net472 2.43μs 1.18ns 4.42ns 0.2 0.00121 0 1.26 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync netcoreapp3.1 1.47μs 0.757ns 2.83ns 0.0178 0 0 1.32 KB
master ExecuteAsync net472 2.42μs 1.03ns 3.99ns 0.219 0.00121 0 1.39 KB
#3816 ExecuteAsync netcoreapp3.1 1.44μs 1.3ns 4.68ns 0.0179 0 0 1.32 KB
#3816 ExecuteAsync net472 2.47μs 0.522ns 1.88ns 0.219 0.00123 0 1.39 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync netcoreapp3.1 4.36μs 1.56ns 5.61ns 0.0413 0 0 3.1 KB
master SendAsync net472 6.89μs 2.52ns 9.07ns 0.541 0 0 3.4 KB
#3816 SendAsync netcoreapp3.1 4.38μs 1.53ns 5.93ns 0.0416 0 0 3.1 KB
#3816 SendAsync net472 6.93μs 2.09ns 8.09ns 0.54 0 0 3.4 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 1.94μs 0.41ns 1.59ns 0.0253 0 0 1.85 KB
master EnrichedLog net472 2.46μs 1.87ns 7.26ns 0.286 0 0 1.81 KB
#3816 EnrichedLog netcoreapp3.1 1.95μs 2.11ns 8.17ns 0.0256 0 0 1.85 KB
#3816 EnrichedLog net472 2.47μs 1.93ns 6.67ns 0.287 0 0 1.81 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 120μs 277ns 1.07μs 0 0 0 4.43 KB
master EnrichedLog net472 151μs 122ns 474ns 0.682 0.227 0 4.65 KB
#3816 EnrichedLog netcoreapp3.1 119μs 290ns 1.12μs 0 0 0 4.43 KB
#3816 EnrichedLog net472 148μs 261ns 1.01μs 0.666 0.222 0 4.65 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 3.93μs 1.45ns 5.24ns 0.0531 0 0 3.91 KB
master EnrichedLog net472 5.12μs 2.03ns 7.31ns 0.567 0.00255 0 3.58 KB
#3816 EnrichedLog netcoreapp3.1 3.83μs 2.11ns 8.19ns 0.0535 0 0 3.91 KB
#3816 EnrichedLog net472 5.09μs 3.34ns 12.9ns 0.567 0.00254 0 3.58 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive netcoreapp3.1 1.58μs 0.701ns 2.72ns 0.0178 0 0 1.3 KB
master SendReceive net472 1.99μs 0.622ns 2.16ns 0.213 0 0 1.34 KB
#3816 SendReceive netcoreapp3.1 1.53μs 0.912ns 3.53ns 0.0176 0 0 1.3 KB
#3816 SendReceive net472 1.94μs 1.73ns 6.69ns 0.212 0 0 1.34 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 3.51μs 1.34ns 5.17ns 0.0228 0 0 1.8 KB
master EnrichedLog net472 4.39μs 2.27ns 8.49ns 0.351 0 0 2.22 KB
#3816 EnrichedLog netcoreapp3.1 3.68μs 1.24ns 4.64ns 0.024 0 0 1.8 KB
#3816 EnrichedLog net472 4.39μs 5.02ns 18.1ns 0.352 0 0 2.22 KB
Benchmarks.Trace.SpanBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan netcoreapp3.1 688ns 0.427ns 1.54ns 0.0101 0 0 736 B
master StartFinishSpan net472 930ns 0.591ns 2.29ns 0.123 0 0 778 B
master StartFinishScope netcoreapp3.1 822ns 0.217ns 0.784ns 0.0115 0 0 856 B
master StartFinishScope net472 1.12μs 0.418ns 1.51ns 0.136 0 0 859 B
#3816 StartFinishSpan netcoreapp3.1 753ns 0.201ns 0.726ns 0.00988 0 0 736 B
#3816 StartFinishSpan net472 913ns 0.512ns 1.98ns 0.124 0 0 778 B
#3816 StartFinishScope netcoreapp3.1 799ns 0.21ns 0.756ns 0.0117 0 0 856 B
#3816 StartFinishScope net472 1.16μs 0.454ns 1.7ns 0.136 0 0 859 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin netcoreapp3.1 968ns 0.475ns 1.84ns 0.0117 0 0 856 B
master RunOnMethodBegin net472 1.25μs 0.548ns 2.05ns 0.136 0 0 859 B
#3816 RunOnMethodBegin netcoreapp3.1 914ns 0.31ns 1.12ns 0.0114 0 0 856 B
#3816 RunOnMethodBegin net472 1.21μs 0.779ns 3.02ns 0.136 0 0 859 B

andrewlock avatar Feb 20 '23 23:02 andrewlock

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,307ms)  : 3242, 3372
     .   : milestone, 3307,
    master - mean (3,324ms)  : 3253, 3395
     .   : milestone, 3324,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,315ms)  : 3232, 3397
     .   : milestone, 3315,
    master - mean (3,323ms)  : 3236, 3411
     .   : milestone, 3323,


gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,189ms)  : 3115, 3262
     .   : milestone, 3189,
    master - mean (3,180ms)  : 3084, 3276
     .   : milestone, 3180,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,198ms)  : 3073, 3323
     .   : milestone, 3198,
    master - mean (3,194ms)  : 3111, 3278
     .   : milestone, 3194,


gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,168ms)  : 3087, 3249
     .   : milestone, 3168,
    master - mean (3,171ms)  : 3115, 3226
     .   : milestone, 3171,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,172ms)  : 3096, 3248
     .   : milestone, 3172,
    master - mean (3,168ms)  : 3102, 3234
     .   : milestone, 3168,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (201ms)  : 189, 213
     .   : milestone, 201,
    master - mean (208ms)  : 193, 222
     .   : milestone, 208,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (202ms)  : 190, 214
     .   : milestone, 202,
    master - mean (205ms)  : 196, 214
     .   : milestone, 205,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (278ms)  : 258, 299
     .   : milestone, 278,
    master - mean (277ms)  : 260, 295
     .   : milestone, 277,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (278ms)  : 258, 298
     .   : milestone, 278,
    master - mean (278ms)  : 256, 300
     .   : milestone, 278,


gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (261ms)  : 243, 280
     .   : milestone, 261,
    master - mean (263ms)  : 243, 282
     .   : milestone, 263,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (262ms)  : 242, 282
     .   : milestone, 262,
    master - mean (265ms)  : 247, 283
     .   : milestone, 265,

andrewlock avatar Feb 20 '23 23:02 andrewlock

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,321ms)  : 3210, 3432
     .   : milestone, 3321,
    master - mean (3,324ms)  : 3253, 3395
     .   : milestone, 3324,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,322ms)  : 3220, 3425
     .   : milestone, 3322,
    master - mean (3,323ms)  : 3236, 3411
     .   : milestone, 3323,


gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,198ms)  : 3124, 3272
     .   : milestone, 3198,
    master - mean (3,180ms)  : 3084, 3276
     .   : milestone, 3180,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,196ms)  : 3111, 3282
     .   : milestone, 3196,
    master - mean (3,194ms)  : 3111, 3278
     .   : milestone, 3194,


gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,170ms)  : 3116, 3224
     .   : milestone, 3170,
    master - mean (3,171ms)  : 3115, 3226
     .   : milestone, 3171,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,169ms)  : 3101, 3236
     .   : milestone, 3169,
    master - mean (3,168ms)  : 3102, 3234
     .   : milestone, 3168,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (205ms)  : 186, 224
     .   : milestone, 205,
    master - mean (208ms)  : 193, 222
     .   : milestone, 208,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (201ms)  : 190, 212
     .   : milestone, 201,
    master - mean (205ms)  : 196, 214
     .   : milestone, 205,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (264ms)  : 248, 281
     .   : milestone, 264,
    master - mean (277ms)  : 260, 295
     .   : milestone, 277,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (270ms)  : 251, 289
     .   : milestone, 270,
    master - mean (278ms)  : 256, 300
     .   : milestone, 278,


gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (262ms)  : 237, 286
     .   : milestone, 262,
    master - mean (263ms)  : 243, 282
     .   : milestone, 263,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (258ms)  : 244, 272
     .   : milestone, 258,
    master - mean (265ms)  : 247, 283
     .   : milestone, 265,

andrewlock avatar Feb 21 '23 00:02 andrewlock

Benchmarks Report :snail:

Benchmarks for #3816 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.182
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces netcoreapp3.1 549μs 579ns 2.24μs 0 0 0 2.62 KB
master WriteAndFlushEnrichedTraces net472 750μs 187ns 673ns 0.379 0 0 3.22 KB
#3816 WriteAndFlushEnrichedTraces netcoreapp3.1 547μs 105ns 379ns 0 0 0 2.63 KB
#3816 WriteAndFlushEnrichedTraces net472 763μs 464ns 1.74μs 0.379 0 0 3.22 KB
Benchmarks.Trace.AppSecBodyBenchmark - Faster :tada: Same allocations :heavy_check_mark:

Faster :tada: in #3816

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑netcoreapp3.1 1.182 281.98 238.49

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody netcoreapp3.1 97.9μs 566ns 4.87μs 0 0 0 1.66 KB
master AllCycleSimpleBody net472 96μs 61.9ns 240ns 0.275 0 0 1.79 KB
master AllCycleMoreComplexBody netcoreapp3.1 297μs 1.72μs 15.1μs 0.155 0 0 12.11 KB
master AllCycleMoreComplexBody net472 293μs 1.1μs 4.25μs 2.01 0 0 13.04 KB
master ObjectExtractorSimpleBody netcoreapp3.1 282ns 0.118ns 0.455ns 0.00368 0 0 272 B
master ObjectExtractorSimpleBody net472 282ns 0.594ns 2.3ns 0.0573 0 0 361 B
master ObjectExtractorMoreComplexBody netcoreapp3.1 13μs 7.03ns 25.3ns 0.0901 0 0 6.75 KB
master ObjectExtractorMoreComplexBody net472 17.8μs 14.3ns 55.3ns 1.21 0.0175 0 7.62 KB
#3816 AllCycleSimpleBody netcoreapp3.1 95.3μs 592ns 5.86μs 0 0 0 1.66 KB
#3816 AllCycleSimpleBody net472 98.1μs 490ns 2.14μs 0.283 0 0 1.79 KB
#3816 AllCycleMoreComplexBody netcoreapp3.1 290μs 1.46μs 7.01μs 0.139 0 0 12.11 KB
#3816 AllCycleMoreComplexBody net472 295μs 128ns 479ns 2.05 0 0 13.04 KB
#3816 ObjectExtractorSimpleBody netcoreapp3.1 238ns 0.0998ns 0.374ns 0.00373 0 0 272 B
#3816 ObjectExtractorSimpleBody net472 279ns 0.0986ns 0.369ns 0.0574 0 0 361 B
#3816 ObjectExtractorMoreComplexBody netcoreapp3.1 12.6μs 6.28ns 23.5ns 0.0884 0 0 6.75 KB
#3816 ObjectExtractorMoreComplexBody net472 16.9μs 24.7ns 95.6ns 1.21 0.0169 0 7.62 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest netcoreapp3.1 182μs 192ns 744ns 0.273 0 0 20.91 KB
master SendRequest net472 0.0389ns 0.000674ns 0.00243ns 0 0 0 0 b
#3816 SendRequest netcoreapp3.1 180μs 261ns 1.01μs 0.271 0 0 20.9 KB
#3816 SendRequest net472 0.0393ns 0.00039ns 0.00146ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces netcoreapp3.1 606μs 692ns 2.68μs 0.592 0 0 41.8 KB
master WriteAndFlushEnrichedTraces net472 774μs 2.54μs 9.83μs 8.33 2.65 0.379 53.23 KB
#3816 WriteAndFlushEnrichedTraces netcoreapp3.1 602μs 1.13μs 4.39μs 0.295 0 0 41.71 KB
#3816 WriteAndFlushEnrichedTraces net472 765μs 1.88μs 7.27μs 8.44 2.68 0.383 53.24 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery netcoreapp3.1 1.26μs 0.448ns 1.74ns 0.0125 0 0 912 B
master ExecuteNonQuery net472 1.54μs 0.43ns 1.67ns 0.145 0 0 915 B
#3816 ExecuteNonQuery netcoreapp3.1 1.26μs 0.814ns 3.05ns 0.0121 0 0 912 B
#3816 ExecuteNonQuery net472 1.62μs 0.375ns 1.45ns 0.145 0 0 915 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch netcoreapp3.1 1.27μs 0.649ns 2.34ns 0.0143 0 0 1.07 KB
master CallElasticsearch net472 2.2μs 1.21ns 4.52ns 0.178 0 0 1.12 KB
master CallElasticsearchAsync netcoreapp3.1 1.38μs 1.19ns 4.61ns 0.0159 0 0 1.19 KB
master CallElasticsearchAsync net472 2.43μs 0.47ns 1.82ns 0.2 0.0012 0 1.26 KB
#3816 CallElasticsearch netcoreapp3.1 1.37μs 0.422ns 1.64ns 0.0143 0 0 1.07 KB
#3816 CallElasticsearch net472 2.28μs 0.908ns 3.4ns 0.178 0 0 1.12 KB
#3816 CallElasticsearchAsync netcoreapp3.1 1.42μs 0.411ns 1.54ns 0.0165 0 0 1.19 KB
#3816 CallElasticsearchAsync net472 2.38μs 0.642ns 2.4ns 0.2 0 0 1.26 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync netcoreapp3.1 1.47μs 0.757ns 2.83ns 0.0178 0 0 1.32 KB
master ExecuteAsync net472 2.42μs 1.03ns 3.99ns 0.219 0.00121 0 1.39 KB
#3816 ExecuteAsync netcoreapp3.1 1.46μs 0.838ns 3.25ns 0.0177 0 0 1.32 KB
#3816 ExecuteAsync net472 2.47μs 1.31ns 4.9ns 0.219 0.00123 0 1.39 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync netcoreapp3.1 4.36μs 1.56ns 5.61ns 0.0413 0 0 3.1 KB
master SendAsync net472 6.89μs 2.52ns 9.07ns 0.541 0 0 3.4 KB
#3816 SendAsync netcoreapp3.1 4.47μs 1.31ns 4.89ns 0.0425 0 0 3.1 KB
#3816 SendAsync net472 6.97μs 2.18ns 8.15ns 0.538 0 0 3.4 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 1.94μs 0.41ns 1.59ns 0.0253 0 0 1.85 KB
master EnrichedLog net472 2.46μs 1.87ns 7.26ns 0.286 0 0 1.81 KB
#3816 EnrichedLog netcoreapp3.1 1.97μs 1.33ns 5.15ns 0.0257 0 0 1.85 KB
#3816 EnrichedLog net472 2.57μs 4.18ns 16.2ns 0.287 0 0 1.81 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 120μs 277ns 1.07μs 0 0 0 4.43 KB
master EnrichedLog net472 151μs 122ns 474ns 0.682 0.227 0 4.65 KB
#3816 EnrichedLog netcoreapp3.1 120μs 158ns 614ns 0.0603 0 0 4.43 KB
#3816 EnrichedLog net472 149μs 244ns 945ns 0.676 0.225 0 4.65 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 3.93μs 1.45ns 5.24ns 0.0531 0 0 3.91 KB
master EnrichedLog net472 5.12μs 2.03ns 7.31ns 0.567 0.00255 0 3.58 KB
#3816 EnrichedLog netcoreapp3.1 3.82μs 2.4ns 8.98ns 0.0534 0 0 3.91 KB
#3816 EnrichedLog net472 5.15μs 4.88ns 18.9ns 0.569 0.00255 0 3.58 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive netcoreapp3.1 1.58μs 0.701ns 2.72ns 0.0178 0 0 1.3 KB
master SendReceive net472 1.99μs 0.622ns 2.16ns 0.213 0 0 1.34 KB
#3816 SendReceive netcoreapp3.1 1.48μs 0.759ns 2.84ns 0.0177 0 0 1.3 KB
#3816 SendReceive net472 1.93μs 0.969ns 3.36ns 0.213 0 0 1.34 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 3.51μs 1.34ns 5.17ns 0.0228 0 0 1.8 KB
master EnrichedLog net472 4.39μs 2.27ns 8.49ns 0.351 0 0 2.22 KB
#3816 EnrichedLog netcoreapp3.1 3.58μs 1.74ns 6.52ns 0.0234 0 0 1.8 KB
#3816 EnrichedLog net472 4.32μs 1.57ns 6.07ns 0.352 0 0 2.22 KB
Benchmarks.Trace.SpanBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan netcoreapp3.1 688ns 0.427ns 1.54ns 0.0101 0 0 736 B
master StartFinishSpan net472 930ns 0.591ns 2.29ns 0.123 0 0 778 B
master StartFinishScope netcoreapp3.1 822ns 0.217ns 0.784ns 0.0115 0 0 856 B
master StartFinishScope net472 1.12μs 0.418ns 1.51ns 0.136 0 0 859 B
#3816 StartFinishSpan netcoreapp3.1 717ns 0.196ns 0.707ns 0.00994 0 0 736 B
#3816 StartFinishSpan net472 949ns 0.254ns 0.917ns 0.123 0 0 778 B
#3816 StartFinishScope netcoreapp3.1 835ns 0.465ns 1.74ns 0.0117 0 0 856 B
#3816 StartFinishScope net472 1.09μs 0.466ns 1.75ns 0.136 0 0 859 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin netcoreapp3.1 968ns 0.475ns 1.84ns 0.0117 0 0 856 B
master RunOnMethodBegin net472 1.25μs 0.548ns 2.05ns 0.136 0 0 859 B
#3816 RunOnMethodBegin netcoreapp3.1 878ns 0.268ns 1.04ns 0.0115 0 0 856 B
#3816 RunOnMethodBegin net472 1.2μs 0.341ns 1.28ns 0.136 0 0 859 B

andrewlock avatar Feb 21 '23 00:02 andrewlock

Benchmarks Report :snail:

Benchmarks for #3816 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.211
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces netcoreapp3.1 549μs 579ns 2.24μs 0 0 0 2.62 KB
master WriteAndFlushEnrichedTraces net472 750μs 187ns 673ns 0.379 0 0 3.22 KB
#3816 WriteAndFlushEnrichedTraces netcoreapp3.1 558μs 226ns 816ns 0 0 0 2.63 KB
#3816 WriteAndFlushEnrichedTraces net472 755μs 318ns 1.19μs 0.383 0 0 3.22 KB
Benchmarks.Trace.AppSecBodyBenchmark - Faster :tada: Same allocations :heavy_check_mark:

Faster :tada: in #3816

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑netcoreapp3.1 1.211 281.98 232.90

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody netcoreapp3.1 97.9μs 566ns 4.87μs 0 0 0 1.66 KB
master AllCycleSimpleBody net472 96μs 61.9ns 240ns 0.275 0 0 1.79 KB
master AllCycleMoreComplexBody netcoreapp3.1 297μs 1.72μs 15.1μs 0.155 0 0 12.11 KB
master AllCycleMoreComplexBody net472 293μs 1.1μs 4.25μs 2.01 0 0 13.04 KB
master ObjectExtractorSimpleBody netcoreapp3.1 282ns 0.118ns 0.455ns 0.00368 0 0 272 B
master ObjectExtractorSimpleBody net472 282ns 0.594ns 2.3ns 0.0573 0 0 361 B
master ObjectExtractorMoreComplexBody netcoreapp3.1 13μs 7.03ns 25.3ns 0.0901 0 0 6.75 KB
master ObjectExtractorMoreComplexBody net472 17.8μs 14.3ns 55.3ns 1.21 0.0175 0 7.62 KB
#3816 AllCycleSimpleBody netcoreapp3.1 99.9μs 556ns 3.52μs 0 0 0 1.66 KB
#3816 AllCycleSimpleBody net472 92.5μs 88.1ns 341ns 0.279 0 0 1.79 KB
#3816 AllCycleMoreComplexBody netcoreapp3.1 282μs 1.36μs 5.6μs 0.143 0 0 12.11 KB
#3816 AllCycleMoreComplexBody net472 306μs 130ns 503ns 2.07 0 0 13.04 KB
#3816 ObjectExtractorSimpleBody netcoreapp3.1 234ns 0.977ns 3.79ns 0.00371 0 0 272 B
#3816 ObjectExtractorSimpleBody net472 292ns 0.15ns 0.559ns 0.0573 0 0 361 B
#3816 ObjectExtractorMoreComplexBody netcoreapp3.1 12.8μs 4.53ns 17ns 0.0891 0 0 6.75 KB
#3816 ObjectExtractorMoreComplexBody net472 16.2μs 16.2ns 62.7ns 1.2 0.0159 0 7.62 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest netcoreapp3.1 182μs 192ns 744ns 0.273 0 0 20.91 KB
master SendRequest net472 0.0389ns 0.000674ns 0.00243ns 0 0 0 0 b
#3816 SendRequest netcoreapp3.1 181μs 172ns 665ns 0.271 0 0 20.9 KB
#3816 SendRequest net472 0.0396ns 0.000549ns 0.00206ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces netcoreapp3.1 606μs 692ns 2.68μs 0.592 0 0 41.8 KB
master WriteAndFlushEnrichedTraces net472 774μs 2.54μs 9.83μs 8.33 2.65 0.379 53.23 KB
#3816 WriteAndFlushEnrichedTraces netcoreapp3.1 607μs 1.39μs 5.38μs 0.308 0 0 41.75 KB
#3816 WriteAndFlushEnrichedTraces net472 771μs 3.64μs 15μs 8.31 2.37 0.396 53.24 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery netcoreapp3.1 1.26μs 0.448ns 1.74ns 0.0125 0 0 912 B
master ExecuteNonQuery net472 1.54μs 0.43ns 1.67ns 0.145 0 0 915 B
#3816 ExecuteNonQuery netcoreapp3.1 1.27μs 0.43ns 1.67ns 0.0121 0 0 912 B
#3816 ExecuteNonQuery net472 1.61μs 0.549ns 2.06ns 0.145 0 0 915 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch netcoreapp3.1 1.27μs 0.649ns 2.34ns 0.0143 0 0 1.07 KB
master CallElasticsearch net472 2.2μs 1.21ns 4.52ns 0.178 0 0 1.12 KB
master CallElasticsearchAsync netcoreapp3.1 1.38μs 1.19ns 4.61ns 0.0159 0 0 1.19 KB
master CallElasticsearchAsync net472 2.43μs 0.47ns 1.82ns 0.2 0.0012 0 1.26 KB
#3816 CallElasticsearch netcoreapp3.1 1.33μs 0.93ns 3.48ns 0.0146 0 0 1.07 KB
#3816 CallElasticsearch net472 2.26μs 1.08ns 4.19ns 0.178 0 0 1.12 KB
#3816 CallElasticsearchAsync netcoreapp3.1 1.35μs 0.471ns 1.76ns 0.0161 0 0 1.19 KB
#3816 CallElasticsearchAsync net472 2.4μs 3.6ns 13.9ns 0.199 0.00119 0 1.26 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync netcoreapp3.1 1.47μs 0.757ns 2.83ns 0.0178 0 0 1.32 KB
master ExecuteAsync net472 2.42μs 1.03ns 3.99ns 0.219 0.00121 0 1.39 KB
#3816 ExecuteAsync netcoreapp3.1 1.48μs 0.595ns 2.23ns 0.0178 0 0 1.32 KB
#3816 ExecuteAsync net472 2.46μs 0.688ns 2.66ns 0.22 0.00121 0 1.39 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync netcoreapp3.1 4.36μs 1.56ns 5.61ns 0.0413 0 0 3.1 KB
master SendAsync net472 6.89μs 2.52ns 9.07ns 0.541 0 0 3.4 KB
#3816 SendAsync netcoreapp3.1 4.42μs 0.665ns 2.4ns 0.0419 0 0 3.1 KB
#3816 SendAsync net472 6.93μs 4.27ns 16.5ns 0.538 0 0 3.4 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 1.94μs 0.41ns 1.59ns 0.0253 0 0 1.85 KB
master EnrichedLog net472 2.46μs 1.87ns 7.26ns 0.286 0 0 1.81 KB
#3816 EnrichedLog netcoreapp3.1 1.95μs 0.389ns 1.51ns 0.0246 0 0 1.85 KB
#3816 EnrichedLog net472 2.54μs 1.92ns 7.42ns 0.286 0 0 1.81 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 120μs 277ns 1.07μs 0 0 0 4.43 KB
master EnrichedLog net472 151μs 122ns 474ns 0.682 0.227 0 4.65 KB
#3816 EnrichedLog netcoreapp3.1 119μs 234ns 904ns 0.0589 0 0 4.43 KB
#3816 EnrichedLog net472 149μs 318ns 1.23μs 0.676 0.225 0 4.65 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 3.93μs 1.45ns 5.24ns 0.0531 0 0 3.91 KB
master EnrichedLog net472 5.12μs 2.03ns 7.31ns 0.567 0.00255 0 3.58 KB
#3816 EnrichedLog netcoreapp3.1 3.96μs 2.04ns 7.63ns 0.0534 0 0 3.91 KB
#3816 EnrichedLog net472 5.09μs 3.49ns 13.5ns 0.567 0.00253 0 3.58 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive netcoreapp3.1 1.58μs 0.701ns 2.72ns 0.0178 0 0 1.3 KB
master SendReceive net472 1.99μs 0.622ns 2.16ns 0.213 0 0 1.34 KB
#3816 SendReceive netcoreapp3.1 1.55μs 1.4ns 5.24ns 0.0178 0 0 1.3 KB
#3816 SendReceive net472 1.89μs 2.35ns 9.1ns 0.213 0.000942 0 1.34 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog netcoreapp3.1 3.51μs 1.34ns 5.17ns 0.0228 0 0 1.8 KB
master EnrichedLog net472 4.39μs 2.27ns 8.49ns 0.351 0 0 2.22 KB
#3816 EnrichedLog netcoreapp3.1 3.69μs 1.01ns 3.76ns 0.0239 0 0 1.8 KB
#3816 EnrichedLog net472 4.41μs 2.23ns 8.36ns 0.352 0 0 2.22 KB
Benchmarks.Trace.SpanBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan netcoreapp3.1 688ns 0.427ns 1.54ns 0.0101 0 0 736 B
master StartFinishSpan net472 930ns 0.591ns 2.29ns 0.123 0 0 778 B
master StartFinishScope netcoreapp3.1 822ns 0.217ns 0.784ns 0.0115 0 0 856 B
master StartFinishScope net472 1.12μs 0.418ns 1.51ns 0.136 0 0 859 B
#3816 StartFinishSpan netcoreapp3.1 692ns 0.963ns 3.73ns 0.01 0 0 736 B
#3816 StartFinishSpan net472 916ns 0.332ns 1.24ns 0.123 0 0 778 B
#3816 StartFinishScope netcoreapp3.1 868ns 0.676ns 2.53ns 0.0114 0 0 856 B
#3816 StartFinishScope net472 1.1μs 0.281ns 1.09ns 0.136 0 0 859 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin netcoreapp3.1 968ns 0.475ns 1.84ns 0.0117 0 0 856 B
master RunOnMethodBegin net472 1.25μs 0.548ns 2.05ns 0.136 0 0 859 B
#3816 RunOnMethodBegin netcoreapp3.1 930ns 0.293ns 1.14ns 0.0116 0 0 856 B
#3816 RunOnMethodBegin net472 1.21μs 0.333ns 1.25ns 0.136 0 0 859 B

andrewlock avatar Feb 21 '23 01:02 andrewlock

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,320ms)  : 3260, 3379
     .   : milestone, 3320,
    master - mean (3,324ms)  : 3253, 3395
     .   : milestone, 3324,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,342ms)  : 3254, 3430
     .   : milestone, 3342,
    master - mean (3,323ms)  : 3236, 3411
     .   : milestone, 3323,


gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,191ms)  : 3107, 3275
     .   : milestone, 3191,
    master - mean (3,180ms)  : 3084, 3276
     .   : milestone, 3180,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,203ms)  : 3085, 3322
     .   : milestone, 3203,
    master - mean (3,194ms)  : 3111, 3278
     .   : milestone, 3194,


gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (3,160ms)  : 3070, 3251
     .   : milestone, 3160,
    master - mean (3,171ms)  : 3115, 3226
     .   : milestone, 3171,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (3,164ms)  : 3092, 3236
     .   : milestone, 3164,
    master - mean (3,168ms)  : 3102, 3234
     .   : milestone, 3168,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (202ms)  : 192, 211
     .   : milestone, 202,
    master - mean (208ms)  : 193, 222
     .   : milestone, 208,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (202ms)  : 186, 219
     .   : milestone, 202,
    master - mean (205ms)  : 196, 214
     .   : milestone, 205,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (268ms)  : 244, 292
     .   : milestone, 268,
    master - mean (277ms)  : 260, 295
     .   : milestone, 277,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (266ms)  : 250, 282
     .   : milestone, 266,
    master - mean (278ms)  : 256, 300
     .   : milestone, 278,


gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (256ms)  : 238, 274
     .   : milestone, 256,
    master - mean (263ms)  : 243, 282
     .   : milestone, 263,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (255ms)  : 242, 268
     .   : milestone, 255,
    master - mean (265ms)  : 247, 283
     .   : milestone, 265,

andrewlock avatar Feb 21 '23 01:02 andrewlock

Code Coverage Report :bar_chart:

:heavy_check_mark: Merging #3816 into master will not change line coverage :heavy_check_mark: Merging #3816 into master will not change branch coverage :heavy_check_mark: Merging #3816 into master will not change complexity

master #3816 Change
Lines 22198 / 31829 22058 / 31829
Lines % 70% 69% 0% :heavy_check_mark:
Branches 13441 / 20889 13368 / 20889
Branches % 64% 64% 0% :heavy_check_mark:
Complexity 22502 22502 0 :heavy_check_mark:

View the full report for further details:

Datadog.Trace Breakdown :heavy_check_mark:

master #3816 Change
Lines % 70% 69% 0% :heavy_check_mark:
Branches % 64% 64% 0% :heavy_check_mark:
Complexity 22502 22502 0 :heavy_check_mark:

The following classes have significant coverage changes.

File Line coverage change Branch coverage change Complexity change
BoundLineProbeLocation -100% :no_entry: 0% :heavy_check_mark: 0 :heavy_check_mark:
Datadog.Trace.Debugger.Models.LineProbeResolveResult -100% :no_entry: 0% :heavy_check_mark: 0 :heavy_check_mark:
Datadog.Trace.Debugger.LineProbeResolver -81% :no_entry: -82% :no_entry: 0 :heavy_check_mark:
Datadog.Trace.Debugger.Configurations.Models.SnapshotSegment -80% :no_entry: -100% :no_entry: 0 :heavy_check_mark:
Datadog.Trace.Pdb.DatadogPdbReader -20% :no_entry: -38% :no_entry: 0 :heavy_check_mark:
Datadog.Trace.Ci.GitInfo -17% :no_entry: -11% :no_entry: 0 :heavy_check_mark:
Datadog.Trace.Debugger.Configurations.Models.Where -8% :no_entry: 0% :heavy_check_mark: 0 :heavy_check_mark:
Datadog.Trace.Ci.Agent.Payloads.EventPlatformPayload -4% :warning: -12% :no_entry: 0 :heavy_check_mark:

View the full reports for further details:

andrewlock avatar Feb 21 '23 02:02 andrewlock

Datadog Report

Branch report: gleocadie/activate-integration-tests-for-arm64 Commit report: 3cc8c5a

:x: dd-trace-dotnet: 36 Failed (1 Known Flaky), 0 New Flaky, 236802 Passed, 785 Skipped, 30m 35.79s Wall Time

:x: Failed Tests (36)

This report shows up to 5 failed tests.

  • UseTracerServiceName - Datadog.Profiler.IntegrationTests.ApplicationInfoTest - Details

    Expand for error 
    xpected infos.Last() to be equal to (BuggyBitsService, BuggyBitsEnv, BuggyBitsVersion), but found (dotnet, Unspecified-Environment, Unspecified-Version).

  • UseTracerServiceName - Datadog.Profiler.IntegrationTests.ApplicationInfoTest - Details

    Expand for error 
    xpected infos.Last() to be equal to (BuggyBitsService, BuggyBitsEnv, BuggyBitsVersion), but found (dotnet, Unspecified-Environment, Unspecified-Version).

  • UseTracerServiceName - Datadog.Profiler.IntegrationTests.ApplicationInfoTest - Details

    Expand for error 
    xpected infos.Last() to be equal to (BuggyBitsService, BuggyBitsEnv, BuggyBitsVersion), but found (dotnet, Unspecified-Environment, Unspecified-Version).

  • CheckEndpointsAreAttached - Datadog.Profiler.IntegrationTests.CodeHotspot.CodeHotspotTest - Details

    Expand for error 
    xit code of "catchsegv" should be 0 instead of 139 (= 0x8B)
xpected: True
ctual:   False

  • CheckEndpointsAreAttached - Datadog.Profiler.IntegrationTests.CodeHotspot.CodeHotspotTest - Details

    Expand for error 
    xpected endpoints.Distinct() to be a collection with 1 item(s), but found an empty collection.

ith configuration:
 Use declared types and members
 Compare enums by value
 Include all non-private properties
 Include all non-private fields
 Match member by name (or throw)
 Without automatic conversion.
 Without automatic conversion.
..

datadog-ddstaging[bot] avatar Mar 01 '23 12:03 datadog-ddstaging[bot]

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (70ms)  : 62, 78
     .   : milestone, 70,
    master - mean (72ms)  : 64, 81
     .   : milestone, 72,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (1,034ms)  : 1009, 1059
     .   : milestone, 1034,
    master - mean (1,034ms)  : 1013, 1056
     .   : milestone, 1034,


gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (106ms)  : 102, 109
     .   : milestone, 106,
    master - mean (105ms)  : 101, 109
     .   : milestone, 105,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (742ms)  : 721, 763
     .   : milestone, 742,
    master - mean (749ms)  : 728, 770
     .   : milestone, 749,


gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (89ms)  : 85, 93
     .   : milestone, 89,
    master - mean (90ms)  : 86, 93
     .   : milestone, 90,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (703ms)  : 686, 720
     .   : milestone, 703,
    master - mean (706ms)  : 684, 727
     .   : milestone, 706,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (191ms)  : 187, 196
     .   : milestone, 191,
    master - mean (191ms)  : 187, 196
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (1,136ms)  : 1099, 1173
     .   : milestone, 1136,
    master - mean (1,140ms)  : 1112, 1168
     .   : milestone, 1140,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (274ms)  : 265, 282
     .   : milestone, 274,
    master - mean (276ms)  : 270, 283
     .   : milestone, 276,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (1,092ms)  : 1065, 1120
     .   : milestone, 1092,
    master - mean (1,106ms)  : 1072, 1140
     .   : milestone, 1106,


gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (3816) - mean (263ms)  : 256, 270
     .   : milestone, 263,
    master - mean (267ms)  : 261, 272
     .   : milestone, 267,

    section CallTarget+Inlining+NGEN
    This PR (3816) - mean (1,050ms)  : 1012, 1088
     .   : milestone, 1050,
    master - mean (1,059ms)  : 1025, 1093
     .   : milestone, 1059,

andrewlock avatar Mar 01 '23 12:03 andrewlock

Benchmarks Report :snail:

Benchmarks for #3816 compared to master:

  • 2 benchmarks are faster, with geometric mean 1.138
  • 1 benchmarks are slower, with geometric mean 1.274
  • 1 benchmarks have fewer allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.11μs 44.6ns 286ns 0.0233 0.0117 0 7.29 KB
master StartStopWithChild netcoreapp3.1 10.2μs 56ns 326ns 0.0289 0.00965 0 7.39 KB
master StartStopWithChild net472 15.7μs 40.4ns 156ns 1.28 0.306 0.102 7.66 KB
#3816 StartStopWithChild net6.0 8.22μs 45.9ns 279ns 0.0212 0.00846 0 7.29 KB
#3816 StartStopWithChild netcoreapp3.1 10.3μs 47.1ns 226ns 0.0301 0.01 0 7.39 KB
#3816 StartStopWithChild net472 15.7μs 42ns 145ns 1.31 0.343 0.112 7.67 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 481μs 263ns 983ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 645μs 135ns 486ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 809μs 187ns 724ns 0.406 0 0 3.3 KB
#3816 WriteAndFlushEnrichedTraces net6.0 479μs 229ns 858ns 0 0 0 2.7 KB
#3816 WriteAndFlushEnrichedTraces netcoreapp3.1 627μs 207ns 802ns 0 0 0 2.7 KB
#3816 WriteAndFlushEnrichedTraces net472 809μs 248ns 929ns 0.403 0 0 3.3 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 42.4μs 30.1ns 109ns 0.0211 0 0 2.03 KB
master AllCycleSimpleBody netcoreapp3.1 45.5μs 73.8ns 276ns 0.0227 0 0 2.01 KB
master AllCycleSimpleBody net472 47.5μs 39.9ns 154ns 0.326 0 0 2.08 KB
master AllCycleMoreComplexBody net6.0 230μs 145ns 563ns 0 0 0 8.63 KB
master AllCycleMoreComplexBody netcoreapp3.1 237μs 263ns 1.02μs 0.118 0 0 8.52 KB
master AllCycleMoreComplexBody net472 239μs 115ns 446ns 1.31 0 0 8.7 KB
master ObjectExtractorSimpleBody net6.0 120ns 0.119ns 0.461ns 0.00393 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 174ns 0.055ns 0.206ns 0.00369 0 0 272 B
master ObjectExtractorSimpleBody net472 149ns 0.134ns 0.517ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 3.07μs 1.64ns 6.12ns 0.0552 0 0 3.88 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 4.09μs 1.86ns 7.2ns 0.0512 0 0 3.78 KB
master ObjectExtractorMoreComplexBody net472 4.18μs 1.32ns 4.94ns 0.617 0.00636 0 3.89 KB
#3816 AllCycleSimpleBody net6.0 42.4μs 14.2ns 54.9ns 0.0212 0 0 2.03 KB
#3816 AllCycleSimpleBody netcoreapp3.1 46.6μs 120ns 450ns 0.0235 0 0 2.01 KB
#3816 AllCycleSimpleBody net472 47.2μs 122ns 471ns 0.323 0 0 2.08 KB
#3816 AllCycleMoreComplexBody net6.0 230μs 84.8ns 317ns 0.115 0 0 8.63 KB
#3816 AllCycleMoreComplexBody netcoreapp3.1 235μs 93.7ns 351ns 0.117 0 0 8.52 KB
#3816 AllCycleMoreComplexBody net472 240μs 94ns 364ns 1.32 0 0 8.7 KB
#3816 ObjectExtractorSimpleBody net6.0 126ns 0.0782ns 0.292ns 0.00396 0 0 280 B
#3816 ObjectExtractorSimpleBody netcoreapp3.1 171ns 0.178ns 0.689ns 0.00371 0 0 272 B
#3816 ObjectExtractorSimpleBody net472 150ns 0.0884ns 0.343ns 0.0446 0 0 281 B
#3816 ObjectExtractorMoreComplexBody net6.0 2.98μs 1.6ns 5.77ns 0.0539 0 0 3.88 KB
#3816 ObjectExtractorMoreComplexBody netcoreapp3.1 4.04μs 1.35ns 5.05ns 0.0506 0 0 3.78 KB
#3816 ObjectExtractorMoreComplexBody net472 4.19μs 1.48ns 5.71ns 0.616 0.00629 0 3.89 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWaf(args=NestedMap (10)) net6.0 12.7μs 5.76ns 21.5ns 0.134 0 0 9.42 KB
master RunWaf(args=NestedMap (10)) netcoreapp3.1 19.2μs 5.7ns 20.6ns 0.125 0 0 9.42 KB
master RunWaf(args=NestedMap (10)) net472 28.1μs 13.2ns 51.1ns 1.5 0 0 9.48 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 78.9μs 232ns 897ns 0.195 0 0 15.77 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 85.8μs 197ns 764ns 0.215 0 0 15.72 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 97.9μs 84.7ns 328ns 2.53 0 0 16.04 KB
master RunWaf(args=NestedMap (100)) net6.0 24.3μs 9.84ns 38.1ns 0.28 0 0 19.66 KB
master RunWaf(args=NestedMap (100)) netcoreapp3.1 37.6μs 18.5ns 69.4ns 0.283 0 0 20.42 KB
master RunWaf(args=NestedMap (100)) net472 54.8μs 16ns 62.1ns 3.26 0.0547 0 20.63 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 92.1μs 39.7ns 149ns 0.361 0 0 26.01 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 109μs 32.5ns 117ns 0.326 0 0 26.72 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net472 131μs 113ns 437ns 4.27 0.0647 0 27.19 KB
master RunWaf(args=NestedMap (20)) net6.0 23.5μs 7.77ns 29.1ns 0.273 0 0 19.4 KB
master RunWaf(args=NestedMap (20)) netcoreapp3.1 38μs 9.96ns 38.6ns 0.266 0 0 19.84 KB
master RunWaf(args=NestedMap (20)) net472 54.7μs 23.3ns 90.4ns 3.16 0.0545 0 20.04 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 94.7μs 109ns 421ns 0.332 0 0 25.74 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 111μs 376ns 1.41μs 0.33 0 0 26.14 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 130μs 113ns 436ns 4.2 0.0646 0 26.6 KB
#3816 RunWaf(args=NestedMap (10)) net6.0 12.8μs 6.73ns 26.1ns 0.128 0 0 9.42 KB
#3816 RunWaf(args=NestedMap (10)) netcoreapp3.1 19.5μs 4.55ns 17.6ns 0.126 0 0 9.42 KB
#3816 RunWaf(args=NestedMap (10)) net472 28μs 11.2ns 43.6ns 1.5 0 0 9.48 KB
#3816 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 77.7μs 29.1ns 113ns 0.197 0 0 15.77 KB
#3816 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 87.9μs 110ns 398ns 0.216 0 0 15.72 KB
#3816 RunWafWithAttack(args=Neste(...)tack) [22]) net472 100μs 56.8ns 213ns 2.5 0 0 16.04 KB
#3816 RunWaf(args=NestedMap (100)) net6.0 25.1μs 11.1ns 41.6ns 0.278 0 0 19.66 KB
#3816 RunWaf(args=NestedMap (100)) netcoreapp3.1 38.3μs 23.1ns 86.5ns 0.268 0 0 20.42 KB
#3816 RunWaf(args=NestedMap (100)) net472 54.1μs 21.4ns 80.2ns 3.27 0.027 0 20.63 KB
#3816 RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 91.3μs 51ns 191ns 0.366 0 0 26.01 KB
#3816 RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 112μs 149ns 577ns 0.335 0 0 26.72 KB
#3816 RunWafWithAttack(args=Neste(...)tack) [23]) net472 130μs 159ns 614ns 4.29 0.065 0 27.19 KB
#3816 RunWaf(args=NestedMap (20)) net6.0 24.5μs 8.86ns 33.1ns 0.271 0 0 19.4 KB
#3816 RunWaf(args=NestedMap (20)) netcoreapp3.1 37.9μs 13.8ns 51.7ns 0.265 0 0 19.84 KB
#3816 RunWaf(args=NestedMap (20)) net472 53.9μs 26.5ns 99.1ns 3.17 0.0538 0 20.04 KB
#3816 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 93μs 53.7ns 208ns 0.368 0 0 25.74 KB
#3816 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 108μs 103ns 385ns 0.326 0 0 26.14 KB
#3816 RunWafWithAttack(args=Neste(...)tack) [22]) net472 132μs 107ns 413ns 4.16 0.065 0 26.6 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 167μs 104ns 390ns 0.167 0 0 18.09 KB
master SendRequest netcoreapp3.1 187μs 302ns 1.17μs 0.188 0 0 20.25 KB
master SendRequest net472 0.000162ns 0.00012ns 0.000463ns 0 0 0 0 b
#3816 SendRequest net6.0 171μs 201ns 780ns 0.256 0 0 18.09 KB
#3816 SendRequest netcoreapp3.1 188μs 165ns 640ns 0.187 0 0 20.25 KB
#3816 SendRequest net472 0.00184ns 0.000382ns 0.00143ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: Fewer allocations :tada:

Fewer allocations :tada: in #3816

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0 41.9 KB 41.38 KB -521 B -1.24%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 544μs 2μs 7.74μs 0.539 0 0 41.9 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 659μs 1.05μs 4.05μs 0.324 0 0 41.62 KB
master WriteAndFlushEnrichedTraces net472 842μs 2.66μs 10.3μs 8.08 2.55 0.425 53.21 KB
#3816 WriteAndFlushEnrichedTraces net6.0 564μs 192ns 743ns 0.546 0 0 41.38 KB
#3816 WriteAndFlushEnrichedTraces netcoreapp3.1 645μs 936ns 3.63μs 0.317 0 0 41.8 KB
#3816 WriteAndFlushEnrichedTraces net472 841μs 3.73μs 14.4μs 8.28 2.48 0.414 53.24 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.03μs 0.684ns 2.65ns 0.0108 0 0 768 B
master ExecuteNonQuery netcoreapp3.1 1.42μs 1.05ns 4.05ns 0.0106 0 0 768 B
master ExecuteNonQuery net472 1.65μs 1.49ns 5.38ns 0.115 0 0 730 B
#3816 ExecuteNonQuery net6.0 1.11μs 0.461ns 1.72ns 0.0105 0 0 768 B
#3816 ExecuteNonQuery netcoreapp3.1 1.33μs 0.925ns 3.58ns 0.00997 0 0 768 B
#3816 ExecuteNonQuery net472 1.66μs 2.3ns 8.89ns 0.115 0 0 730 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.16μs 5.33ns 20.6ns 0.0138 0 0 992 B
master CallElasticsearch netcoreapp3.1 1.48μs 0.581ns 2.01ns 0.0133 0 0 992 B
master CallElasticsearch net472 2.35μs 0.402ns 1.5ns 0.158 0 0 1 KB
master CallElasticsearchAsync net6.0 1.27μs 0.659ns 2.47ns 0.0133 0 0 968 B
master CallElasticsearchAsync netcoreapp3.1 1.5μs 0.563ns 2.18ns 0.0142 0 0 1.04 KB
master CallElasticsearchAsync net472 2.42μs 0.787ns 3.05ns 0.167 0 0 1.06 KB
#3816 CallElasticsearch net6.0 1.28μs 1.02ns 3.81ns 0.014 0 0 992 B
#3816 CallElasticsearch netcoreapp3.1 1.48μs 0.557ns 2.08ns 0.0134 0 0 992 B
#3816 CallElasticsearch net472 2.42μs 1.39ns 5.36ns 0.159 0 0 1 KB
#3816 CallElasticsearchAsync net6.0 1.24μs 2.06ns 7.7ns 0.0135 0 0 968 B
#3816 CallElasticsearchAsync netcoreapp3.1 1.64μs 0.429ns 1.49ns 0.0139 0 0 1.04 KB
#3816 CallElasticsearchAsync net472 2.55μs 1.04ns 3.88ns 0.168 0 0 1.06 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.23μs 6.26ns 28.7ns 0.0129 0 0 912 B
master ExecuteAsync netcoreapp3.1 1.48μs 0.865ns 3.35ns 0.0126 0 0 912 B
master ExecuteAsync net472 1.68μs 0.828ns 3.1ns 0.138 0 0 875 B
#3816 ExecuteAsync net6.0 1.3μs 0.442ns 1.65ns 0.0123 0 0 912 B
#3816 ExecuteAsync netcoreapp3.1 1.49μs 1.4ns 5.05ns 0.0125 0 0 912 B
#3816 ExecuteAsync net472 1.77μs 0.739ns 2.86ns 0.138 0 0 875 B
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 3.75μs 1.6ns 6ns 0.0277 0 0 1.94 KB
master SendAsync netcoreapp3.1 4.66μs 18ns 69.7ns 0.0343 0 0 2.48 KB
master SendAsync net472 7.1μs 1.47ns 5.29ns 0.481 0 0 3.05 KB
#3816 SendAsync net6.0 3.77μs 1.25ns 4.67ns 0.0265 0 0 1.94 KB
#3816 SendAsync netcoreapp3.1 4.48μs 1.45ns 5.02ns 0.0342 0 0 2.48 KB
#3816 SendAsync net472 7.09μs 1.72ns 6.64ns 0.484 0 0 3.05 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.29μs 0.714ns 2.76ns 0.0226 0 0 1.62 KB
master EnrichedLog netcoreapp3.1 1.97μs 0.799ns 3.09ns 0.0217 0 0 1.62 KB
master EnrichedLog net472 2.42μs 2.28ns 8.54ns 0.245 0 0 1.54 KB
#3816 EnrichedLog net6.0 1.37μs 0.733ns 2.84ns 0.0226 0 0 1.62 KB
#3816 EnrichedLog netcoreapp3.1 1.93μs 0.842ns 3.04ns 0.0217 0 0 1.62 KB
#3816 EnrichedLog net472 2.43μs 1.46ns 5.26ns 0.244 0 0 1.54 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 113μs 152ns 588ns 0.0559 0 0 4.21 KB
master EnrichedLog netcoreapp3.1 120μs 155ns 579ns 0.0596 0 0 4.21 KB
master EnrichedLog net472 148μs 92.2ns 357ns 0.669 0.223 0 4.38 KB
#3816 EnrichedLog net6.0 115μs 102ns 395ns 0.0578 0 0 4.21 KB
#3816 EnrichedLog netcoreapp3.1 116μs 117ns 454ns 0 0 0 4.21 KB
#3816 EnrichedLog net472 147μs 58.6ns 211ns 0.664 0.221 0 4.38 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.84μs 0.771ns 2.88ns 0.0311 0 0 2.18 KB
master EnrichedLog netcoreapp3.1 3.9μs 1.5ns 5.61ns 0.0292 0 0 2.18 KB
master EnrichedLog net472 4.6μs 1.98ns 7.65ns 0.316 0 0 1.99 KB
#3816 EnrichedLog net6.0 2.91μs 8.42ns 32.6ns 0.0303 0 0 2.18 KB
#3816 EnrichedLog netcoreapp3.1 3.82μs 1.99ns 7.72ns 0.0287 0 0 2.18 KB
#3816 EnrichedLog net472 4.66μs 3.66ns 14.2ns 0.316 0 0 1.99 KB
Benchmarks.Trace.RedisBenchmark - Faster :tada: Same allocations :heavy_check_mark:

Faster :tada: in #3816

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.RedisBenchmark.SendReceive‑net6.0 1.146 1,420.79 1,239.82

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.43μs 3.09ns 12ns 0.0164 0 0 1.16 KB
master SendReceive netcoreapp3.1 1.62μs 0.623ns 2.41ns 0.0154 0 0 1.16 KB
master SendReceive net472 2.2μs 1.78ns 6.18ns 0.185 0 0 1.16 KB
#3816 SendReceive net6.0 1.24μs 0.926ns 3.47ns 0.016 0 0 1.16 KB
#3816 SendReceive netcoreapp3.1 1.63μs 4.39ns 17ns 0.0161 0 0 1.16 KB
#3816 SendReceive net472 2.03μs 1.22ns 4.39ns 0.185 0 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.55μs 0.845ns 3.27ns 0.0217 0 0 1.53 KB
master EnrichedLog netcoreapp3.1 3.65μs 1.99ns 7.45ns 0.0201 0 0 1.58 KB
master EnrichedLog net472 4.04μs 1.25ns 4.52ns 0.309 0 0 1.96 KB
#3816 EnrichedLog net6.0 2.64μs 1.1ns 4.26ns 0.0211 0 0 1.53 KB
#3816 EnrichedLog netcoreapp3.1 3.58μs 17.5ns 70.2ns 0.0213 0 0 1.58 KB
#3816 EnrichedLog net472 4.08μs 2.17ns 8.41ns 0.31 0 0 1.96 KB
Benchmarks.Trace.SpanBenchmark - Slower :warning: Same allocations :heavy_check_mark:

Slower :warning: in #3816

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0 1.274 471.86 601.29

Faster :tada: in #3816

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 1.131 502.67 444.62

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 503ns 0.33ns 1.28ns 0.00734 0 0 536 B
master StartFinishSpan netcoreapp3.1 633ns 0.253ns 0.945ns 0.00734 0 0 536 B
master StartFinishSpan net472 608ns 0.163ns 0.611ns 0.0854 0 0 538 B
master StartFinishScope net6.0 472ns 0.399ns 1.49ns 0.00926 0 0 656 B
master StartFinishScope netcoreapp3.1 744ns 0.4ns 1.55ns 0.00866 0 0 656 B
master StartFinishScope net472 855ns 0.416ns 1.61ns 0.0982 0 0 618 B
#3816 StartFinishSpan net6.0 445ns 0.161ns 0.625ns 0.00747 0 0 536 B
#3816 StartFinishSpan netcoreapp3.1 659ns 3.44ns 16.9ns 0.00715 0 0 536 B
#3816 StartFinishSpan net472 654ns 0.159ns 0.616ns 0.0854 0 0 538 B
#3816 StartFinishScope net6.0 601ns 0.205ns 0.794ns 0.00904 0 0 656 B
#3816 StartFinishScope netcoreapp3.1 721ns 0.329ns 1.27ns 0.00905 0 0 656 B
#3816 StartFinishScope net472 795ns 0.598ns 2.31ns 0.098 0 0 618 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 645ns 0.186ns 0.72ns 0.00927 0 0 656 B
master RunOnMethodBegin netcoreapp3.1 771ns 2.55ns 9.89ns 0.00871 0 0 656 B
master RunOnMethodBegin net472 936ns 0.325ns 1.22ns 0.098 0 0 618 B
#3816 RunOnMethodBegin net6.0 670ns 0.259ns 0.97ns 0.00898 0 0 656 B
#3816 RunOnMethodBegin netcoreapp3.1 753ns 0.258ns 1ns 0.00901 0 0 656 B
#3816 RunOnMethodBegin net472 966ns 0.291ns 1.13ns 0.0979 0 0 618 B

andrewlock avatar Mar 01 '23 13:03 andrewlock

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (3816) (11.051M)   : 0, 11050811
    master (10.984M)   : 0, 10984393
    benchmarks/2.38.0 (11.076M)   : 0, 11076318
    benchmarks/2.9.0 (11.262M)   : 0, 11262462

    section Automatic
    This PR (3816) (7.752M)   : 0, 7751847
    master (7.613M)   : 0, 7613182
    benchmarks/2.38.0 (7.662M)   : 0, 7661825
    benchmarks/2.9.0 (8.118M)   : 0, 8118446

    section Trace stats
    master (7.907M)   : 0, 7907173
    benchmarks/2.38.0 (8.254M)   : 0, 8253906

    section Manual
    This PR (3816) (9.956M)   : 0, 9955622
    master (9.541M)   : 0, 9540645
    benchmarks/2.38.0 (10.086M)   : 0, 10085508

    section Manual + Automatic
    This PR (3816) (7.362M)   : 0, 7361587
    master (7.109M)   : 0, 7109106
    benchmarks/2.38.0 (7.551M)   : 0, 7550577

    section Version Conflict
    master (6.637M)   : 0, 6637256
    benchmarks/2.38.0 (6.926M)   : 0, 6925707


gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (3816) (9.547M)   : 0, 9546932
    master (9.081M)   : 0, 9081380
    benchmarks/2.38.0 (9.450M)   : 0, 9450019
    benchmarks/2.9.0 (9.656M)   : 0, 9656397

    section Automatic
    This PR (3816) (6.880M)   : 0, 6879516
    master (6.476M)   : 0, 6476412
    benchmarks/2.38.0 (6.565M)   : 0, 6564966

    section Trace stats
    master (7.049M)   : 0, 7049333
    benchmarks/2.38.0 (6.919M)   : 0, 6918614

    section Manual
    This PR (3816) (8.415M)   : 0, 8414839
    master (8.574M)   : 0, 8573609
    benchmarks/2.38.0 (8.471M)   : 0, 8471172

    section Manual + Automatic
    This PR (3816) (6.216M)   : 0, 6215805
    master (6.253M)   : 0, 6253237
    benchmarks/2.38.0 (6.145M)   : 0, 6144975

    section Version Conflict
    master (5.849M)   : 0, 5849359
    benchmarks/2.38.0 (5.806M)   : 0, 5806248


gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (3816) (10.263M)   : 0, 10263346
    master (10.259M)   : 0, 10259098
    benchmarks/2.38.0 (10.454M)   : 0, 10454111
    benchmarks/2.9.0 (10.115M)   : 0, 10114719

    section Automatic
    This PR (3816) (7.160M)   : 0, 7159908
    master (7.259M)   : 0, 7259474
    benchmarks/2.38.0 (7.348M)   : 0, 7347757
    benchmarks/2.9.0 (7.385M)   : 0, 7384780

    section Trace stats
    master (7.463M)   : 0, 7462678
    benchmarks/2.38.0 (7.583M)   : 0, 7583441

    section Manual
    This PR (3816) (9.065M)   : 0, 9064584
    master (9.119M)   : 0, 9119393
    benchmarks/2.38.0 (9.403M)   : 0, 9402906

    section Manual + Automatic
    This PR (3816) (6.981M)   : 0, 6981345
    master (7.066M)   : 0, 7065522
    benchmarks/2.38.0 (7.018M)   : 0, 7018067

    section Version Conflict
    master (6.433M)   : 0, 6432546
    benchmarks/2.38.0 (6.405M)   : 0, 6405209


gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (3816) (7.554M)   : 0, 7554399
    master (7.571M)   : 0, 7570884
    benchmarks/2.38.0 (7.398M)   : 0, 7398154
    benchmarks/2.9.0 (7.750M)   : 0, 7750019

    section No attack
    This PR (3816) (2.161M)   : 0, 2160770
    master (2.157M)   : 0, 2156696
    benchmarks/2.38.0 (2.150M)   : 0, 2150468
    benchmarks/2.9.0 (3.195M)   : 0, 3195477

    section Attack
    This PR (3816) (1.732M)   : 0, 1732144
    master (1.727M)   : 0, 1727208
    benchmarks/2.38.0 (1.713M)   : 0, 1712973
    benchmarks/2.9.0 (2.529M)   : 0, 2528837

    section Blocking
    This PR (3816) (3.473M)   : 0, 3472675
    master (3.489M)   : 0, 3489355
    benchmarks/2.38.0 (3.405M)   : 0, 3405427

andrewlock avatar Mar 01 '23 15:03 andrewlock

Datadog Report

Branch report: gleocadie/activate-integration-tests-for-arm64 Commit report: 2fe7077

:x: dd-trace-dotnet: 4 Failed (0 Known Flaky), 0 New Flaky, 306002 Passed, 1199 Skipped, 21m 32.29s Wall Time

:x: Failed Tests (4)

  • CheckCpuLimit - Datadog.Profiler.IntegrationTests.SmokeTests.CpuLimitTest - Details

    Expand for error 
    xit code of "catchsegv" should be 0 instead of 139 (= 0x8B)
xpected: True
ctual:   False

  • CheckCpuLimit - Datadog.Profiler.IntegrationTests.SmokeTests.CpuLimitTest - Details

    Expand for error 
    xit code of "catchsegv" should be 0 instead of 139 (= 0x8B)
xpected: True
ctual:   False

  • CheckCpuLimit - Datadog.Profiler.IntegrationTests.SmokeTests.CpuLimitTest - Details

    Expand for error 
    xit code of "catchsegv" should be 0 instead of 139 (= 0x8B)
xpected: True
ctual:   False

  • WhenUsingUdsAgent_UsesUdsTelemetry - Datadog.Trace.ClrProfiler.IntegrationTests.TelemetryTests - Details

    Expand for error 
    xpected relevantSpans to contain at least 3 item(s) because we want to ensure that we don't timeout while waiting for spans from the mock tracer agent, but found 2: {TraceId: 13108737201910270656, SpanId: 8948367441963234429, Name: http.request, Resource: GET localhost:54670/?/, Service: Samples.Telemetry-http-client, TraceId: 13108737201910270656, SpanId: 2658948518606519617, Name: GetAsync, Resource: GetAsync, Service: Samples.Telemetry}.

datadog-ddstaging[bot] avatar Sep 26 '23 17:09 datadog-ddstaging[bot]

Datadog Report

Branch report: gleocadie/activate-integration-tests-for-arm64 Commit report: 09c2d01

:x: dd-trace-dotnet: 1 Failed (0 Known Flaky), 0 New Flaky, 306551 Passed, 1203 Skipped, 23m 18.82s Wall Time

:x: Failed Tests (1)

  • CheckCpuLimit - Datadog.Profiler.IntegrationTests.SmokeTests.CpuLimitTest - Details

    Expand for error 
    xpected agent.NbCallsOnProfilingEndpoint to be greater than 0, but found 0.

datadog-ddstaging[bot] avatar Oct 03 '23 17:10 datadog-ddstaging[bot]