dd-trace-dotnet icon indicating copy to clipboard operation
dd-trace-dotnet copied to clipboard
verified publisher icon DataDog

[Dynamic Instrumentation] Added support for line instrumentation in the exploration tests

Open GreenMatan opened this issue 2 years ago • 6 comments

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

GreenMatan avatar Oct 24 '23 09:10 GreenMatan

Datadog Report

Branch report: matang/line-exploration-test-support Commit report: 4b523e5

:x: dd-trace-dotnet: 25 Failed (0 Known Flaky), 0 New Flaky, 188259 Passed, 1315 Skipped, 55m 42.85s Wall Time

:x: Failed Tests (25)

This report shows up to 5 failed tests.

  • AllConfigurationValuesAreRegisteredWithIntake - Datadog.Trace.Tests.Telemetry.ConfigurationTests - Details

    Expand for error 
    xpected collection to be empty because Keys should be listed in config_norm_rules or block_prefixes, but found {"DD_INTERNAL_DEBUGGER_INSTRUMENT_ALL_LINES", "DD_INTERNAL_DEBUGGER_INSTRUMENT_ALL_LINES_PATH"}.

  • AllConfigurationValuesAreRegisteredWithIntake - Datadog.Trace.Tests.Telemetry.ConfigurationTests - Details

    Expand for error 
    xpected collection to be empty because Keys should be listed in config_norm_rules or block_prefixes, but found {"DD_INTERNAL_DEBUGGER_INSTRUMENT_ALL_LINES", "DD_INTERNAL_DEBUGGER_INSTRUMENT_ALL_LINES_PATH"}.

  • AllConfigurationValuesAreRegisteredWithIntake - Datadog.Trace.Tests.Telemetry.ConfigurationTests - Details

    Expand for error 
    xpected collection to be empty because Keys should be listed in config_norm_rules or block_prefixes, but found {"DD_INTERNAL_DEBUGGER_INSTRUMENT_ALL_LINES", "DD_INTERNAL_DEBUGGER_INSTRUMENT_ALL_LINES_PATH"}.

  • AllConfigurationValuesAreRegisteredWithIntake - Datadog.Trace.Tests.Telemetry.ConfigurationTests - Details

    Expand for error 
    xpected collection to be empty because Keys should be listed in config_norm_rules or block_prefixes, but found {"DD_INTERNAL_DEBUGGER_INSTRUMENT_ALL_LINES", "DD_INTERNAL_DEBUGGER_INSTRUMENT_ALL_LINES_PATH"}.

  • AllConfigurationValuesAreRegisteredWithIntake - Datadog.Trace.Tests.Telemetry.ConfigurationTests - Details

    Expand for error 
    xpected collection to be empty because Keys should be listed in config_norm_rules or block_prefixes, but found {"DD_INTERNAL_DEBUGGER_INSTRUMENT_ALL_LINES", "DD_INTERNAL_DEBUGGER_INSTRUMENT_ALL_LINES_PATH"}.

datadog-ddstaging[bot] avatar Oct 24 '23 09:10 datadog-ddstaging[bot]

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4754) - mean (74ms)  : 65, 83
     .   : milestone, 74,
    master - mean (75ms)  : 65, 84
     .   : milestone, 75,

    section CallTarget+Inlining+NGEN
    This PR (4754) - mean (1,014ms)  : 990, 1037
     .   : milestone, 1014,
    master - mean (1,009ms)  : 980, 1038
     .   : milestone, 1009,


gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4754) - mean (111ms)  : 107, 115
     .   : milestone, 111,
    master - mean (110ms)  : 107, 113
     .   : milestone, 110,

    section CallTarget+Inlining+NGEN
    This PR (4754) - mean (730ms)  : 698, 762
     .   : milestone, 730,
    master - mean (732ms)  : 704, 760
     .   : milestone, 732,


gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4754) - mean (94ms)  : 91, 96
     .   : milestone, 94,
    master - mean (95ms)  : 92, 99
     .   : milestone, 95,

    section CallTarget+Inlining+NGEN
    This PR (4754) - mean (682ms)  : 662, 703
     .   : milestone, 682,
    master - mean (683ms)  : 653, 713
     .   : milestone, 683,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4754) - mean (192ms)  : 188, 195
     .   : milestone, 192,
    master - mean (192ms)  : 188, 196
     .   : milestone, 192,

    section CallTarget+Inlining+NGEN
    This PR (4754) - mean (1,116ms)  : 1092, 1141
     .   : milestone, 1116,
    master - mean (1,110ms)  : 1092, 1128
     .   : milestone, 1110,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4754) - mean (275ms)  : 269, 281
     .   : milestone, 275,
    master - mean (275ms)  : 271, 280
     .   : milestone, 275,

    section CallTarget+Inlining+NGEN
    This PR (4754) - mean (913ms)  : 895, 931
     .   : milestone, 913,
    master - mean (914ms)  : 897, 930
     .   : milestone, 914,


gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4754) - mean (266ms)  : 263, 269
     .   : milestone, 266,
    master - mean (265ms)  : 260, 270
     .   : milestone, 265,

    section CallTarget+Inlining+NGEN
    This PR (4754) - mean (894ms)  : 871, 918
     .   : milestone, 894,
    master - mean (894ms)  : 870, 919
     .   : milestone, 894,

andrewlock avatar Oct 24 '23 10:10 andrewlock

Benchmarks Report :snail:

Benchmarks for #4754 compared to master:

  • 3 benchmarks are faster, with geometric mean 1.168
  • 1 benchmarks are slower, with geometric mean 1.165
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.21μs 45ns 281ns 0.0244 0.00815 0 7.29 KB
master StartStopWithChild netcoreapp3.1 10.1μs 56.3ns 374ns 0.0304 0.0152 0 7.38 KB
master StartStopWithChild net472 15.7μs 61.3ns 237ns 1.29 0.329 0.0919 7.67 KB
#4754 StartStopWithChild net6.0 8.26μs 44.4ns 243ns 0.0284 0.0122 0 7.29 KB
#4754 StartStopWithChild netcoreapp3.1 9.86μs 55.5ns 360ns 0.0299 0.015 0 7.39 KB
#4754 StartStopWithChild net472 15.5μs 50.4ns 195ns 1.28 0.318 0.0853 7.67 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 488μs 252ns 976ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 638μs 718ns 2.59μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 795μs 550ns 2.06μs 0.398 0 0 3.3 KB
#4754 WriteAndFlushEnrichedTraces net6.0 462μs 251ns 971ns 0 0 0 2.7 KB
#4754 WriteAndFlushEnrichedTraces netcoreapp3.1 636μs 286ns 1.11μs 0 0 0 2.7 KB
#4754 WriteAndFlushEnrichedTraces net472 800μs 471ns 1.63μs 0.396 0 0 3.3 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Faster :tada: Same allocations :heavy_check_mark:

Faster :tada: in #4754

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑net6.0 1.240 156.04 125.89

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 41.8μs 18.6ns 72.1ns 0.0209 0 0 2.03 KB
master AllCycleSimpleBody netcoreapp3.1 45.3μs 99.2ns 371ns 0.0224 0 0 2.01 KB
master AllCycleSimpleBody net472 46.8μs 40.2ns 156ns 0.325 0 0 2.08 KB
master AllCycleMoreComplexBody net6.0 230μs 90.4ns 350ns 0.115 0 0 8.63 KB
master AllCycleMoreComplexBody netcoreapp3.1 233μs 118ns 456ns 0.119 0 0 8.52 KB
master AllCycleMoreComplexBody net472 239μs 111ns 431ns 1.31 0 0 8.7 KB
master ObjectExtractorSimpleBody net6.0 156ns 0.0384ns 0.144ns 0.00394 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 173ns 0.223ns 0.865ns 0.00375 0 0 272 B
master ObjectExtractorSimpleBody net472 150ns 0.0829ns 0.31ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 3.07μs 0.943ns 3.53ns 0.0536 0 0 3.88 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 4.08μs 2.15ns 8.32ns 0.0514 0 0 3.78 KB
master ObjectExtractorMoreComplexBody net472 4.1μs 1.99ns 7.71ns 0.617 0.00617 0 3.89 KB
#4754 AllCycleSimpleBody net6.0 43.3μs 25.1ns 97.4ns 0.0217 0 0 2.03 KB
#4754 AllCycleSimpleBody netcoreapp3.1 46.1μs 55.8ns 216ns 0.0229 0 0 2.01 KB
#4754 AllCycleSimpleBody net472 46.9μs 78ns 292ns 0.308 0 0 2.08 KB
#4754 AllCycleMoreComplexBody net6.0 229μs 62.3ns 233ns 0.114 0 0 8.63 KB
#4754 AllCycleMoreComplexBody netcoreapp3.1 235μs 121ns 468ns 0.117 0 0 8.52 KB
#4754 AllCycleMoreComplexBody net472 239μs 113ns 436ns 1.31 0 0 8.7 KB
#4754 ObjectExtractorSimpleBody net6.0 126ns 0.115ns 0.447ns 0.00393 0 0 280 B
#4754 ObjectExtractorSimpleBody netcoreapp3.1 176ns 0.0779ns 0.302ns 0.00364 0 0 272 B
#4754 ObjectExtractorSimpleBody net472 150ns 0.13ns 0.502ns 0.0446 0 0 281 B
#4754 ObjectExtractorMoreComplexBody net6.0 3.01μs 0.815ns 3.05ns 0.0544 0 0 3.88 KB
#4754 ObjectExtractorMoreComplexBody netcoreapp3.1 4.03μs 1.33ns 4.98ns 0.0505 0 0 3.78 KB
#4754 ObjectExtractorMoreComplexBody net472 4.14μs 3.51ns 13.6ns 0.617 0.00619 0 3.89 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWaf(args=NestedMap (10)) net6.0 12.8μs 3.74ns 14ns 0.134 0 0 9.42 KB
master RunWaf(args=NestedMap (10)) netcoreapp3.1 19.7μs 12.5ns 48.5ns 0.127 0 0 9.42 KB
master RunWaf(args=NestedMap (10)) net472 28.4μs 10.6ns 39.6ns 1.49 0 0 9.48 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 75.9μs 41.1ns 154ns 0.195 0 0 15.77 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 86.6μs 148ns 574ns 0.175 0 0 15.72 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 97.1μs 44.1ns 159ns 2.53 0 0 16.04 KB
master RunWaf(args=NestedMap (100)) net6.0 24.2μs 9.61ns 36ns 0.268 0 0 19.66 KB
master RunWaf(args=NestedMap (100)) netcoreapp3.1 38.2μs 17.1ns 66.3ns 0.268 0 0 20.42 KB
master RunWaf(args=NestedMap (100)) net472 54.8μs 19.5ns 75.4ns 3.25 0.0274 0 20.63 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 92.8μs 31.5ns 122ns 0.37 0 0 26.01 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 111μs 50.3ns 181ns 0.33 0 0 26.72 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net472 130μs 54.6ns 197ns 4.29 0.065 0 27.19 KB
master RunWaf(args=NestedMap (20)) net6.0 25.2μs 7.11ns 27.5ns 0.265 0 0 19.4 KB
master RunWaf(args=NestedMap (20)) netcoreapp3.1 37.8μs 12.4ns 48ns 0.264 0 0 19.84 KB
master RunWaf(args=NestedMap (20)) net472 53.5μs 22.1ns 85.7ns 3.18 0.0534 0 20.04 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 90.8μs 31.6ns 122ns 0.362 0 0 25.74 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 108μs 171ns 662ns 0.321 0 0 26.14 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 129μs 56.8ns 205ns 4.21 0.0647 0 26.6 KB
#4754 RunWaf(args=NestedMap (10)) net6.0 13.7μs 4.69ns 17.5ns 0.13 0 0 9.42 KB
#4754 RunWaf(args=NestedMap (10)) netcoreapp3.1 19.4μs 7.37ns 27.6ns 0.126 0 0 9.42 KB
#4754 RunWaf(args=NestedMap (10)) net472 28.1μs 9.44ns 36.6ns 1.5 0 0 9.48 KB
#4754 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 76.6μs 46.3ns 173ns 0.228 0 0 15.77 KB
#4754 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 87.7μs 146ns 548ns 0.177 0 0 15.72 KB
#4754 RunWafWithAttack(args=Neste(...)tack) [22]) net472 98.8μs 208ns 807ns 2.53 0 0 16.04 KB
#4754 RunWaf(args=NestedMap (100)) net6.0 23.9μs 7.86ns 29.4ns 0.275 0 0 19.66 KB
#4754 RunWaf(args=NestedMap (100)) netcoreapp3.1 37.7μs 11.5ns 43.1ns 0.264 0 0 20.42 KB
#4754 RunWaf(args=NestedMap (100)) net472 55.4μs 21.1ns 81.7ns 3.26 0.0276 0 20.63 KB
#4754 RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 94.9μs 21.1ns 78.9ns 0.332 0 0 26.01 KB
#4754 RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 110μs 160ns 619ns 0.329 0 0 26.72 KB
#4754 RunWafWithAttack(args=Neste(...)tack) [23]) net472 134μs 63.8ns 247ns 4.27 0.0667 0 27.19 KB
#4754 RunWaf(args=NestedMap (20)) net6.0 24.5μs 4.87ns 18.2ns 0.27 0 0 19.4 KB
#4754 RunWaf(args=NestedMap (20)) netcoreapp3.1 37.9μs 12.7ns 47.4ns 0.266 0 0 19.84 KB
#4754 RunWaf(args=NestedMap (20)) net472 53.6μs 12.5ns 46.9ns 3.16 0.0536 0 20.04 KB
#4754 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 90.9μs 121ns 470ns 0.362 0 0 25.74 KB
#4754 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 109μs 92.9ns 360ns 0.33 0 0 26.14 KB
#4754 RunWafWithAttack(args=Neste(...)tack) [22]) net472 130μs 91.1ns 328ns 4.19 0.0654 0 26.6 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 167μs 226ns 877ns 0.166 0 0 18.04 KB
master SendRequest netcoreapp3.1 190μs 267ns 1.04μs 0.189 0 0 20.2 KB
master SendRequest net472 0.000613ns 0.000219ns 0.000847ns 0 0 0 0 b
#4754 SendRequest net6.0 168μs 138ns 534ns 0.167 0 0 18.04 KB
#4754 SendRequest netcoreapp3.1 191μs 308ns 1.19μs 0.191 0 0 20.2 KB
#4754 SendRequest net472 0.000754ns 0.000249ns 0.00093ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 533μs 322ns 1.25μs 0.539 0 0 41.38 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 662μs 1.74μs 6.72μs 0.329 0 0 41.78 KB
master WriteAndFlushEnrichedTraces net472 869μs 4.11μs 15.9μs 8.19 2.59 0.431 53.25 KB
#4754 WriteAndFlushEnrichedTraces net6.0 545μs 700ns 2.71μs 0.561 0 0 41.46 KB
#4754 WriteAndFlushEnrichedTraces netcoreapp3.1 650μs 1.5μs 5.82μs 0.327 0 0 41.62 KB
#4754 WriteAndFlushEnrichedTraces net472 831μs 4.24μs 18.9μs 8.22 2.47 0.411 53.24 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.03μs 0.92ns 3.56ns 0.0104 0 0 768 B
master ExecuteNonQuery netcoreapp3.1 1.25μs 2.13ns 8.24ns 0.0105 0 0 768 B
master ExecuteNonQuery net472 1.61μs 0.509ns 1.84ns 0.116 0 0 730 B
#4754 ExecuteNonQuery net6.0 1.02μs 1.51ns 5.85ns 0.0107 0 0 768 B
#4754 ExecuteNonQuery netcoreapp3.1 1.3μs 0.618ns 2.31ns 0.0103 0 0 768 B
#4754 ExecuteNonQuery net472 1.75μs 0.701ns 2.62ns 0.116 0 0 730 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.08μs 0.566ns 2.19ns 0.0131 0 0 936 B
master CallElasticsearch netcoreapp3.1 1.38μs 2.39ns 9.24ns 0.0126 0 0 936 B
master CallElasticsearch net472 2.48μs 0.762ns 2.95ns 0.152 0 0 955 B
master CallElasticsearchAsync net6.0 1.37μs 1.36ns 5.25ns 0.0124 0 0 912 B
master CallElasticsearchAsync netcoreapp3.1 1.49μs 0.711ns 2.56ns 0.0132 0 0 984 B
master CallElasticsearchAsync net472 2.56μs 0.939ns 3.64ns 0.16 0 0 1.01 KB
#4754 CallElasticsearch net6.0 1.17μs 0.343ns 1.33ns 0.0128 0 0 936 B
#4754 CallElasticsearch netcoreapp3.1 1.38μs 0.93ns 3.48ns 0.0123 0 0 936 B
#4754 CallElasticsearch net472 2.33μs 1.06ns 4.1ns 0.151 0 0 955 B
#4754 CallElasticsearchAsync net6.0 1.24μs 1.04ns 3.88ns 0.0123 0 0 912 B
#4754 CallElasticsearchAsync netcoreapp3.1 1.52μs 0.471ns 1.82ns 0.0136 0 0 984 B
#4754 CallElasticsearchAsync net472 2.58μs 1.5ns 5.81ns 0.16 0 0 1.01 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.24μs 1.34ns 5.2ns 0.0125 0 0 912 B
master ExecuteAsync netcoreapp3.1 1.55μs 0.909ns 3.52ns 0.0124 0 0 912 B
master ExecuteAsync net472 1.7μs 0.595ns 2.22ns 0.139 0.000851 0 875 B
#4754 ExecuteAsync net6.0 1.17μs 3.47ns 13.4ns 0.0127 0 0 912 B
#4754 ExecuteAsync netcoreapp3.1 1.45μs 0.372ns 1.34ns 0.0125 0 0 912 B
#4754 ExecuteAsync net472 1.65μs 0.74ns 2.87ns 0.138 0 0 875 B
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 3.84μs 3.88ns 15ns 0.0271 0 0 1.9 KB
master SendAsync netcoreapp3.1 4.34μs 1.99ns 7.19ns 0.0324 0 0 2.43 KB
master SendAsync net472 7.06μs 1.28ns 4.62ns 0.475 0 0 2.99 KB
#4754 SendAsync net6.0 3.66μs 1.25ns 4.67ns 0.0254 0 0 1.9 KB
#4754 SendAsync netcoreapp3.1 4.42μs 1.63ns 6.09ns 0.0334 0 0 2.43 KB
#4754 SendAsync net472 7.1μs 2.15ns 7.76ns 0.476 0 0 2.99 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.34μs 0.547ns 2.05ns 0.022 0 0 1.57 KB
master EnrichedLog netcoreapp3.1 1.93μs 0.624ns 2.34ns 0.0212 0 0 1.57 KB
master EnrichedLog net472 2.33μs 1.31ns 5.09ns 0.236 0 0 1.49 KB
#4754 EnrichedLog net6.0 1.49μs 0.677ns 2.44ns 0.0217 0 0 1.57 KB
#4754 EnrichedLog netcoreapp3.1 1.86μs 0.741ns 2.87ns 0.0215 0 0 1.57 KB
#4754 EnrichedLog net472 2.24μs 0.884ns 3.06ns 0.236 0 0 1.49 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 111μs 79.6ns 298ns 0.0559 0 0 4.21 KB
master EnrichedLog netcoreapp3.1 116μs 120ns 448ns 0 0 0 4.21 KB
master EnrichedLog net472 146μs 108ns 418ns 0.658 0.219 0 4.38 KB
#4754 EnrichedLog net6.0 112μs 150ns 561ns 0.0553 0 0 4.21 KB
#4754 EnrichedLog netcoreapp3.1 120μs 65ns 243ns 0 0 0 4.21 KB
#4754 EnrichedLog net472 146μs 61.7ns 239ns 0.662 0.221 0 4.38 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.06μs 0.948ns 3.55ns 0.0305 0 0 2.13 KB
master EnrichedLog netcoreapp3.1 3.95μs 2.31ns 8.96ns 0.0293 0 0 2.13 KB
master EnrichedLog net472 4.7μs 1.64ns 6.12ns 0.305 0 0 1.93 KB
#4754 EnrichedLog net6.0 2.86μs 0.84ns 3.14ns 0.03 0 0 2.13 KB
#4754 EnrichedLog netcoreapp3.1 4.05μs 1.44ns 5.39ns 0.0284 0 0 2.13 KB
#4754 EnrichedLog net472 4.64μs 1.49ns 5.78ns 0.305 0 0 1.93 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.29μs 0.895ns 3.47ns 0.0155 0 0 1.1 KB
master SendReceive netcoreapp3.1 1.65μs 0.564ns 2.03ns 0.015 0 0 1.1 KB
master SendReceive net472 1.98μs 0.691ns 2.59ns 0.177 0 0 1.12 KB
#4754 SendReceive net6.0 1.25μs 0.752ns 2.81ns 0.0156 0 0 1.1 KB
#4754 SendReceive netcoreapp3.1 1.66μs 0.641ns 2.4ns 0.0149 0 0 1.1 KB
#4754 SendReceive net472 1.94μs 1.29ns 4.48ns 0.177 0 0 1.12 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.46μs 0.522ns 2.02ns 0.0209 0 0 1.53 KB
master EnrichedLog netcoreapp3.1 3.5μs 1.66ns 6.41ns 0.021 0 0 1.58 KB
master EnrichedLog net472 4.11μs 1.51ns 5.87ns 0.31 0 0 1.96 KB
#4754 EnrichedLog net6.0 2.64μs 0.978ns 3.79ns 0.0211 0 0 1.53 KB
#4754 EnrichedLog netcoreapp3.1 3.45μs 0.968ns 3.35ns 0.0209 0 0 1.58 KB
#4754 EnrichedLog net472 4μs 1.79ns 6.94ns 0.31 0 0 1.96 KB
Benchmarks.Trace.SpanBenchmark - Slower :warning: Same allocations :heavy_check_mark:

Slower :warning: in #4754

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 1.165 381.15 443.88

Faster :tada: in #4754

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net472 1.153 917.79 795.91
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1 1.114 758.15 680.57

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 381ns 0.163ns 0.61ns 0.00757 0 0 536 B
master StartFinishSpan netcoreapp3.1 535ns 0.206ns 0.771ns 0.00725 0 0 536 B
master StartFinishSpan net472 583ns 0.161ns 0.622ns 0.0852 0 0 538 B
master StartFinishScope net6.0 591ns 0.263ns 0.948ns 0.00918 0 0 656 B
master StartFinishScope netcoreapp3.1 758ns 0.339ns 1.31ns 0.00875 0 0 656 B
master StartFinishScope net472 918ns 0.28ns 1.01ns 0.0981 0 0 618 B
#4754 StartFinishSpan net6.0 444ns 0.147ns 0.567ns 0.00739 0 0 536 B
#4754 StartFinishSpan netcoreapp3.1 494ns 0.179ns 0.694ns 0.00741 0 0 536 B
#4754 StartFinishSpan net472 611ns 0.153ns 0.572ns 0.0852 0 0 538 B
#4754 StartFinishScope net6.0 583ns 0.141ns 0.545ns 0.009 0 0 656 B
#4754 StartFinishScope netcoreapp3.1 680ns 0.337ns 1.26ns 0.00891 0 0 656 B
#4754 StartFinishScope net472 795ns 0.444ns 1.72ns 0.098 0 0 618 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 591ns 0.227ns 0.878ns 0.00926 0 0 656 B
master RunOnMethodBegin netcoreapp3.1 799ns 2.17ns 7.5ns 0.00893 0 0 656 B
master RunOnMethodBegin net472 1.04μs 0.368ns 1.42ns 0.0977 0 0 618 B
#4754 RunOnMethodBegin net6.0 609ns 0.213ns 0.825ns 0.00938 0 0 656 B
#4754 RunOnMethodBegin netcoreapp3.1 842ns 1.29ns 4.99ns 0.00881 0 0 656 B
#4754 RunOnMethodBegin net472 1.05μs 0.58ns 2.25ns 0.098 0 0 618 B

andrewlock avatar Oct 24 '23 13:10 andrewlock

Datadog Report

Branch report: matang/line-exploration-test-support Commit report: ad4a892 Test service: dd-trace-dotnet

:x: 25 Failed (0 Known Flaky), 280274 Passed, 1987 Skipped, 21h 15m 13.2s Total Time :hourglass: 2 Performance Regressions

:x: Failed Tests (25)

This report shows up to 5 failed tests.

  • AllConfigurationValuesAreRegisteredWithIntake - Datadog.Trace.Tests.Telemetry.ConfigurationTests - Details

    Expand for error 
    xpected collection to be empty because Keys should be listed in config_norm_rules or block_prefixes, but found {"DD_INTERNAL_DEBUGGER_INSTRUMENT_ALL_LINES", "DD_INTERNAL_DEBUGGER_INSTRUMENT_ALL_LINES_PATH"}.

  • AllConfigurationValuesAreRegisteredWithIntake - Datadog.Trace.Tests.Telemetry.ConfigurationTests - Details

    Expand for error 
    xpected collection to be empty because Keys should be listed in config_norm_rules or block_prefixes, but found {"DD_INTERNAL_DEBUGGER_INSTRUMENT_ALL_LINES", "DD_INTERNAL_DEBUGGER_INSTRUMENT_ALL_LINES_PATH"}.

  • AllConfigurationValuesAreRegisteredWithIntake - Datadog.Trace.Tests.Telemetry.ConfigurationTests - Details

    Expand for error 
    xpected collection to be empty because Keys should be listed in config_norm_rules or block_prefixes, but found {"DD_INTERNAL_DEBUGGER_INSTRUMENT_ALL_LINES", "DD_INTERNAL_DEBUGGER_INSTRUMENT_ALL_LINES_PATH"}.

  • AllConfigurationValuesAreRegisteredWithIntake - Datadog.Trace.Tests.Telemetry.ConfigurationTests - Details

    Expand for error 
    xpected collection to be empty because Keys should be listed in config_norm_rules or block_prefixes, but found {"DD_INTERNAL_DEBUGGER_INSTRUMENT_ALL_LINES", "DD_INTERNAL_DEBUGGER_INSTRUMENT_ALL_LINES_PATH"}.

  • AllConfigurationValuesAreRegisteredWithIntake - Datadog.Trace.Tests.Telemetry.ConfigurationTests - Details

    Expand for error 
    xpected collection to be empty because Keys should be listed in config_norm_rules or block_prefixes, but found {"DD_INTERNAL_DEBUGGER_INSTRUMENT_ALL_LINES", "DD_INTERNAL_DEBUGGER_INSTRUMENT_ALL_LINES_PATH"}.

:hourglass: Performance Regressions vs Default Branch (2)

  • Profiler_exceptions - scenarios 5.14s (+785.2ms, +18%) - Details
  • Baseline_exception - scenarios 3.94s (+137.32ms, +4%) - Details

datadog-ddstaging[bot] avatar Mar 26 '24 14:03 datadog-ddstaging[bot]

Benchmarks Report for tracer :snail:

Benchmarks for #4754 compared to master:

  • 2 benchmarks are faster, with geometric mean 1.216
  • 1 benchmarks are slower, with geometric mean 1.251
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.8μs 48.6ns 315ns 0.0218 0.00873 0 7.56 KB
master StartStopWithChild netcoreapp3.1 10.6μs 49.8ns 193ns 0.0215 0.0107 0 7.66 KB
master StartStopWithChild net472 17.1μs 56.2ns 218ns 1.33 0.342 0.102 8.06 KB
#4754 StartStopWithChild net6.0 8.6μs 47.5ns 300ns 0.0209 0.00835 0 7.55 KB
#4754 StartStopWithChild netcoreapp3.1 10.3μs 52.9ns 270ns 0.0256 0.0103 0 7.65 KB
#4754 StartStopWithChild net472 17.1μs 31.8ns 115ns 1.35 0.37 0.109 8.06 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 507μs 168ns 629ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 622μs 459ns 1.78μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 823μs 404ns 1.51μs 0.411 0 0 3.3 KB
#4754 WriteAndFlushEnrichedTraces net6.0 476μs 140ns 524ns 0 0 0 2.7 KB
#4754 WriteAndFlushEnrichedTraces netcoreapp3.1 634μs 318ns 1.23μs 0 0 0 2.7 KB
#4754 WriteAndFlushEnrichedTraces net472 833μs 415ns 1.61μs 0.414 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 170μs 95ns 355ns 0.169 0 0 18.49 KB
master SendRequest netcoreapp3.1 192μs 270ns 1.05μs 0.192 0 0 20.65 KB
master SendRequest net472 0.000792ns 0.00041ns 0.00153ns 0 0 0 0 b
#4754 SendRequest net6.0 169μs 214ns 828ns 0.17 0 0 18.49 KB
#4754 SendRequest netcoreapp3.1 191μs 304ns 1.18μs 0.189 0 0 20.65 KB
#4754 SendRequest net472 0.00117ns 0.000285ns 0.00107ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 547μs 301ns 1.17μs 0.548 0 0 41.71 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 650μs 2.49μs 9.66μs 0.324 0 0 41.69 KB
master WriteAndFlushEnrichedTraces net472 818μs 3.69μs 14.3μs 8.22 2.47 0.411 53.24 KB
#4754 WriteAndFlushEnrichedTraces net6.0 547μs 1.17μs 4.54μs 0.541 0 0 41.68 KB
#4754 WriteAndFlushEnrichedTraces netcoreapp3.1 657μs 951ns 3.56μs 0.329 0 0 41.64 KB
#4754 WriteAndFlushEnrichedTraces net472 867μs 3.9μs 23.1μs 8.08 2.55 0.425 53.21 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.09μs 0.739ns 2.86ns 0.0114 0 0 808 B
master ExecuteNonQuery netcoreapp3.1 1.5μs 0.948ns 3.67ns 0.0111 0 0 808 B
master ExecuteNonQuery net472 1.77μs 0.424ns 1.47ns 0.122 0 0 770 B
#4754 ExecuteNonQuery net6.0 1.08μs 0.494ns 1.91ns 0.0115 0 0 808 B
#4754 ExecuteNonQuery netcoreapp3.1 1.52μs 2.22ns 8.62ns 0.0106 0 0 808 B
#4754 ExecuteNonQuery net472 1.79μs 1.01ns 3.91ns 0.122 0 0 770 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.25μs 0.668ns 2.59ns 0.0143 0 0 1.03 KB
master CallElasticsearch netcoreapp3.1 1.65μs 1.63ns 6.31ns 0.0141 0 0 1.03 KB
master CallElasticsearch net472 2.62μs 0.951ns 3.68ns 0.166 0 0 1.04 KB
master CallElasticsearchAsync net6.0 1.28μs 0.6ns 2.32ns 0.0141 0 0 1.01 KB
master CallElasticsearchAsync netcoreapp3.1 1.7μs 1.31ns 5.06ns 0.0145 0 0 1.08 KB
master CallElasticsearchAsync net472 2.55μs 0.849ns 3.29ns 0.175 0 0 1.1 KB
#4754 CallElasticsearch net6.0 1.25μs 0.47ns 1.76ns 0.0145 0 0 1.03 KB
#4754 CallElasticsearch netcoreapp3.1 1.63μs 0.293ns 1.14ns 0.0139 0 0 1.03 KB
#4754 CallElasticsearch net472 2.51μs 0.843ns 3.26ns 0.165 0 0 1.04 KB
#4754 CallElasticsearchAsync net6.0 1.25μs 0.945ns 3.54ns 0.0139 0 0 1.01 KB
#4754 CallElasticsearchAsync netcoreapp3.1 1.63μs 1.73ns 6.24ns 0.0146 0 0 1.08 KB
#4754 CallElasticsearchAsync net472 2.61μs 1.63ns 5.64ns 0.174 0 0 1.1 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.28μs 0.925ns 3.46ns 0.0134 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.64μs 0.676ns 2.62ns 0.0124 0 0 952 B
master ExecuteAsync net472 1.77μs 0.66ns 2.56ns 0.145 0.000883 0 915 B
#4754 ExecuteAsync net6.0 1.34μs 0.979ns 3.66ns 0.0134 0 0 952 B
#4754 ExecuteAsync netcoreapp3.1 1.66μs 3.39ns 13.1ns 0.0133 0 0 952 B
#4754 ExecuteAsync net472 1.86μs 1ns 3.89ns 0.145 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.11μs 1.74ns 6.75ns 0.0307 0 0 2.27 KB
master SendAsync netcoreapp3.1 5.08μs 3.47ns 12.5ns 0.0381 0 0 2.81 KB
master SendAsync net472 7.64μs 2.72ns 10.5ns 0.503 0 0 3.18 KB
#4754 SendAsync net6.0 4.17μs 5.09ns 19.7ns 0.0314 0 0 2.27 KB
#4754 SendAsync netcoreapp3.1 5.12μs 4.58ns 17.7ns 0.0358 0 0 2.81 KB
#4754 SendAsync net472 7.81μs 5.99ns 23.2ns 0.505 0 0 3.18 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.58μs 1.08ns 4.03ns 0.0238 0 0 1.7 KB
master EnrichedLog netcoreapp3.1 2.23μs 1.18ns 4.41ns 0.0235 0 0 1.7 KB
master EnrichedLog net472 2.69μs 1.62ns 6.26ns 0.257 0 0 1.62 KB
#4754 EnrichedLog net6.0 1.45μs 0.725ns 2.81ns 0.0239 0 0 1.7 KB
#4754 EnrichedLog netcoreapp3.1 2.15μs 0.851ns 3.3ns 0.0225 0 0 1.7 KB
#4754 EnrichedLog net472 2.62μs 1.89ns 7.06ns 0.257 0 0 1.62 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 113μs 86.4ns 323ns 0.0562 0 0 4.28 KB
master EnrichedLog netcoreapp3.1 117μs 119ns 459ns 0.0586 0 0 4.28 KB
master EnrichedLog net472 147μs 50.9ns 190ns 0.659 0.22 0 4.46 KB
#4754 EnrichedLog net6.0 113μs 70.6ns 264ns 0.0564 0 0 4.28 KB
#4754 EnrichedLog netcoreapp3.1 117μs 89.2ns 346ns 0 0 0 4.28 KB
#4754 EnrichedLog net472 147μs 63.4ns 237ns 0.663 0.221 0 4.46 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.88μs 0.727ns 2.72ns 0.0306 0 0 2.25 KB
master EnrichedLog netcoreapp3.1 4.25μs 1.88ns 7.04ns 0.0299 0 0 2.25 KB
master EnrichedLog net472 4.88μs 1.54ns 5.78ns 0.328 0 0 2.07 KB
#4754 EnrichedLog net6.0 3.1μs 1.76ns 6.59ns 0.0311 0 0 2.25 KB
#4754 EnrichedLog netcoreapp3.1 4.18μs 5.25ns 19.6ns 0.0308 0 0 2.25 KB
#4754 EnrichedLog net472 4.82μs 2.55ns 9.87ns 0.328 0 0 2.07 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.35μs 0.964ns 3.61ns 0.0168 0 0 1.2 KB
master SendReceive netcoreapp3.1 1.82μs 3.32ns 12.4ns 0.0163 0 0 1.2 KB
master SendReceive net472 2.25μs 1.42ns 5.33ns 0.19 0.00113 0 1.2 KB
#4754 SendReceive net6.0 1.42μs 0.547ns 2.12ns 0.017 0 0 1.2 KB
#4754 SendReceive netcoreapp3.1 1.87μs 0.55ns 2.13ns 0.0159 0 0 1.2 KB
#4754 SendReceive net472 2.22μs 1.63ns 6.33ns 0.191 0 0 1.2 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.88μs 3.25ns 12.2ns 0.0216 0 0 1.6 KB
master EnrichedLog netcoreapp3.1 4.18μs 4.84ns 18.8ns 0.0223 0 0 1.65 KB
master EnrichedLog net472 4.44μs 1.57ns 6.09ns 0.322 0 0 2.04 KB
#4754 EnrichedLog net6.0 2.62μs 0.521ns 1.95ns 0.0223 0 0 1.6 KB
#4754 EnrichedLog netcoreapp3.1 4.05μs 3.65ns 14.2ns 0.0223 0 0 1.65 KB
#4754 EnrichedLog net472 4.44μs 1.45ns 5.22ns 0.323 0 0 2.04 KB
Benchmarks.Trace.SpanBenchmark - Slower :warning: Same allocations :heavy_check_mark:

Slower :warning: in #4754

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 1.251 464.59 581.16

Faster :tada: in #4754

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472 1.297 892.17 687.99
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1 1.140 909.98 798.38

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 465ns 0.144ns 0.558ns 0.00814 0 0 576 B
master StartFinishSpan netcoreapp3.1 736ns 2.96ns 11.5ns 0.00758 0 0 576 B
master StartFinishSpan net472 892ns 0.204ns 0.764ns 0.0914 0 0 578 B
master StartFinishScope net6.0 626ns 0.293ns 1.1ns 0.0098 0 0 696 B
master StartFinishScope netcoreapp3.1 906ns 1.76ns 6.8ns 0.00952 0 0 696 B
master StartFinishScope net472 1.07μs 0.694ns 2.69ns 0.104 0 0 658 B
#4754 StartFinishSpan net6.0 581ns 0.205ns 0.767ns 0.00809 0 0 576 B
#4754 StartFinishSpan netcoreapp3.1 695ns 1.53ns 5.93ns 0.00798 0 0 576 B
#4754 StartFinishSpan net472 689ns 0.63ns 2.44ns 0.0917 0 0 578 B
#4754 StartFinishScope net6.0 588ns 0.251ns 0.938ns 0.00974 0 0 696 B
#4754 StartFinishScope netcoreapp3.1 802ns 1.51ns 5.84ns 0.00938 0 0 696 B
#4754 StartFinishScope net472 963ns 1.1ns 4.12ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 607ns 0.185ns 0.715ns 0.00994 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 931ns 1.33ns 5.14ns 0.00914 0 0 696 B
master RunOnMethodBegin net472 1.13μs 0.343ns 1.33ns 0.104 0 0 658 B
#4754 RunOnMethodBegin net6.0 605ns 0.271ns 1.05ns 0.00977 0 0 696 B
#4754 RunOnMethodBegin netcoreapp3.1 922ns 0.845ns 3.27ns 0.00922 0 0 696 B
#4754 RunOnMethodBegin net472 1.18μs 0.703ns 2.72ns 0.104 0 0 658 B

andrewlock avatar Mar 26 '24 15:03 andrewlock

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4754) (12.137M)   : 0, 12136851
    master (11.879M)   : 0, 11879110
    benchmarks/2.9.0 (11.983M)   : 0, 11982622

    section Automatic
    This PR (4754) (8.041M)   : 0, 8040920
    master (8.054M)   : 0, 8053530
    benchmarks/2.9.0 (8.324M)   : 0, 8324349

    section Trace stats
    master (8.362M)   : 0, 8361687

    section Manual
    This PR (4754) (10.336M)   : 0, 10335649
    master (10.229M)   : 0, 10229080

    section Manual + Automatic
    This PR (4754) (7.447M)   : 0, 7447168
    master (7.641M)   : 0, 7641349

    section Version Conflict
    master (6.825M)   : 0, 6824800


gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4754) (9.467M)   : 0, 9466844
    master (9.599M)   : 0, 9598656
    benchmarks/2.9.0 (9.587M)   : 0, 9587201

    section Automatic
    This PR (4754) (6.621M)   : 0, 6620813
    master (6.588M)   : 0, 6588326

    section Trace stats
    master (6.893M)   : 0, 6892670

    section Manual
    This PR (4754) (8.293M)   : 0, 8293407
    master (8.131M)   : 0, 8130516

    section Manual + Automatic
    This PR (4754) (6.241M)   : 0, 6241346
    master (6.167M)   : 0, 6167127

    section Version Conflict
    master (5.576M)   : 0, 5575919


gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4754) (9.879M)   : 0, 9879298
    master (9.987M)   : 0, 9986822
    benchmarks/2.9.0 (9.913M)   : 0, 9912627

    section Automatic
    This PR (4754) (7.062M)   : 0, 7062129
    master (6.986M)   : 0, 6985911
    benchmarks/2.9.0 (7.312M)   : 0, 7312269

    section Trace stats
    master (7.286M)   : 0, 7285868

    section Manual
    This PR (4754) (8.761M)   : 0, 8761218
    master (8.789M)   : 0, 8788786

    section Manual + Automatic
    This PR (4754) (6.876M)   : 0, 6876189
    master (6.745M)   : 0, 6745109

    section Version Conflict
    master (6.204M)   : 0, 6204499

andrewlock avatar Apr 09 '24 09:04 andrewlock