dd-trace-dotnet icon indicating copy to clipboard operation
dd-trace-dotnet copied to clipboard
verified publisher icon DataDog

[Dynamic Instrumentation] DEBUG-2088 Support nullable types in templates

Open dudikeleti opened this issue 1 year ago • 5 comments

Summary of changes

Support the usage of nullable types (i.e. Guid?) in the DI Expression Language

Test coverage

See tests in the PR

dudikeleti avatar May 08 '24 16:05 dudikeleti

Datadog Report

Branch report: dudik/el-nullable-type Commit report: dc8508f Test service: dd-trace-dotnet

:white_check_mark: 0 Failed, 335784 Passed, 2034 Skipped, 21h 30m 7.96s Total Time

datadog-ddstaging[bot] avatar May 08 '24 16:05 datadog-ddstaging[bot]

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5543) - mean (73ms)  : 63, 83
     .   : milestone, 73,
    master - mean (73ms)  : 65, 82
     .   : milestone, 73,

    section CallTarget+Inlining+NGEN
    This PR (5543) - mean (1,069ms)  : 1044, 1094
     .   : milestone, 1069,
    master - mean (1,079ms)  : 1057, 1100
     .   : milestone, 1079,


gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5543) - mean (109ms)  : 105, 114
     .   : milestone, 109,
    master - mean (110ms)  : 105, 114
     .   : milestone, 110,

    section CallTarget+Inlining+NGEN
    This PR (5543) - mean (752ms)  : 730, 773
     .   : milestone, 752,
    master - mean (761ms)  : 732, 790
     .   : milestone, 761,


gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5543) - mean (92ms)  : 90, 95
     .   : milestone, 92,
    master - mean (93ms)  : 90, 95
     .   : milestone, 93,

    section CallTarget+Inlining+NGEN
    This PR (5543) - mean (703ms)  : 686, 721
     .   : milestone, 703,
    master - mean (712ms)  : 687, 738
     .   : milestone, 712,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5543) - mean (192ms)  : 189, 195
     .   : milestone, 192,
    master - mean (192ms)  : 189, 195
     .   : milestone, 192,

    section CallTarget+Inlining+NGEN
    This PR (5543) - mean (1,170ms)  : 1144, 1196
     .   : milestone, 1170,
    master - mean (1,170ms)  : 1150, 1190
     .   : milestone, 1170,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5543) - mean (277ms)  : 272, 281
     .   : milestone, 277,
    master - mean (277ms)  : 273, 281
     .   : milestone, 277,

    section CallTarget+Inlining+NGEN
    This PR (5543) - mean (919ms)  : 899, 939
     .   : milestone, 919,
    master - mean (923ms)  : 899, 946
     .   : milestone, 923,


gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5543) - mean (266ms)  : 261, 272
     .   : milestone, 266,
    master - mean (265ms)  : 261, 269
     .   : milestone, 265,

    section CallTarget+Inlining+NGEN
    This PR (5543) - mean (904ms)  : 885, 923
     .   : milestone, 904,
    master - mean (910ms)  : 888, 931
     .   : milestone, 910,

andrewlock avatar May 08 '24 16:05 andrewlock

Benchmarks Report for tracer :snail:

Benchmarks for #5543 compared to master:

  • 1 benchmarks are slower, with geometric mean 1.142
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 7.78μs 41.8ns 304ns 0.015 0.0075 0 5.43 KB
master StartStopWithChild netcoreapp3.1 9.96μs 53.9ns 310ns 0.0192 0.00962 0 5.62 KB
master StartStopWithChild net472 15.9μs 45ns 168ns 1.02 0.293 0.1 6.06 KB
#5543 StartStopWithChild net6.0 7.75μs 44.3ns 332ns 0.0153 0.00765 0 5.43 KB
#5543 StartStopWithChild netcoreapp3.1 9.87μs 42.3ns 179ns 0.0198 0.00992 0 5.61 KB
#5543 StartStopWithChild net472 16.2μs 66ns 256ns 1.03 0.31 0.0953 6.06 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 441μs 379ns 1.47μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 625μs 463ns 1.79μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 851μs 337ns 1.22μs 0.425 0 0 3.3 KB
#5543 WriteAndFlushEnrichedTraces net6.0 481μs 1.42μs 5.51μs 0 0 0 2.7 KB
#5543 WriteAndFlushEnrichedTraces netcoreapp3.1 645μs 522ns 1.95μs 0 0 0 2.7 KB
#5543 WriteAndFlushEnrichedTraces net472 861μs 666ns 2.49μs 0.425 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 197μs 1.15μs 10μs 0.194 0 0 18.45 KB
master SendRequest netcoreapp3.1 221μs 1.24μs 7.93μs 0.208 0 0 20.61 KB
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#5543 SendRequest net6.0 196μs 1.11μs 8.74μs 0.191 0 0 18.45 KB
#5543 SendRequest netcoreapp3.1 214μs 1.2μs 8.06μs 0.223 0 0 20.61 KB
#5543 SendRequest net472 0.00281ns 0.00113ns 0.00439ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 582μs 2.94μs 13.8μs 0.573 0 0 41.52 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 683μs 3.75μs 22.2μs 0.324 0 0 41.93 KB
master WriteAndFlushEnrichedTraces net472 853μs 4.08μs 16.8μs 8.33 2.5 0.417 53.28 KB
#5543 WriteAndFlushEnrichedTraces net6.0 602μs 3.37μs 21.3μs 0.566 0 0 41.7 KB
#5543 WriteAndFlushEnrichedTraces netcoreapp3.1 706μs 3.61μs 18.1μs 0.338 0 0 41.88 KB
#5543 WriteAndFlushEnrichedTraces net472 874μs 4.36μs 20μs 8.19 2.59 0.431 53.27 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.25μs 1.32ns 4.94ns 0.0143 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 1.76μs 2.79ns 10.8ns 0.0131 0 0 1.02 KB
master ExecuteNonQuery net472 2.01μs 1.23ns 4.28ns 0.156 0 0 987 B
#5543 ExecuteNonQuery net6.0 1.2μs 2.19ns 8.49ns 0.0146 0 0 1.02 KB
#5543 ExecuteNonQuery netcoreapp3.1 1.75μs 1.12ns 4.34ns 0.0131 0 0 1.02 KB
#5543 ExecuteNonQuery net472 2.01μs 1.65ns 6.4ns 0.156 0 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.14μs 0.551ns 2.06ns 0.0137 0 0 976 B
master CallElasticsearch netcoreapp3.1 1.51μs 1.46ns 5.27ns 0.0129 0 0 976 B
master CallElasticsearch net472 2.47μs 2.52ns 9.75ns 0.158 0 0 995 B
master CallElasticsearchAsync net6.0 1.24μs 0.384ns 1.44ns 0.0137 0 0 952 B
master CallElasticsearchAsync netcoreapp3.1 1.64μs 1.24ns 4.64ns 0.014 0 0 1.02 KB
master CallElasticsearchAsync net472 2.69μs 1.66ns 6.42ns 0.167 0 0 1.05 KB
#5543 CallElasticsearch net6.0 1.22μs 0.601ns 2.25ns 0.0135 0 0 976 B
#5543 CallElasticsearch netcoreapp3.1 1.46μs 0.587ns 2.2ns 0.0132 0 0 976 B
#5543 CallElasticsearch net472 2.42μs 2.79ns 10.8ns 0.157 0 0 995 B
#5543 CallElasticsearchAsync net6.0 1.25μs 0.707ns 2.65ns 0.0132 0 0 952 B
#5543 CallElasticsearchAsync netcoreapp3.1 1.64μs 1.12ns 4.18ns 0.0139 0 0 1.02 KB
#5543 CallElasticsearchAsync net472 2.66μs 2.09ns 8.1ns 0.166 0 0 1.05 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.24μs 1.12ns 4.33ns 0.0136 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.56μs 0.597ns 2.31ns 0.0125 0 0 952 B
master ExecuteAsync net472 1.77μs 0.499ns 1.87ns 0.145 0 0 915 B
#5543 ExecuteAsync net6.0 1.16μs 1.04ns 3.88ns 0.0134 0 0 952 B
#5543 ExecuteAsync netcoreapp3.1 1.58μs 0.555ns 2.08ns 0.0126 0 0 952 B
#5543 ExecuteAsync net472 1.7μs 0.5ns 1.94ns 0.145 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.17μs 1.97ns 7.65ns 0.0314 0 0 2.22 KB
master SendAsync netcoreapp3.1 5.11μs 1.45ns 5.44ns 0.0359 0 0 2.76 KB
master SendAsync net472 7.86μs 8.74ns 33.8ns 0.497 0 0 3.15 KB
#5543 SendAsync net6.0 4.23μs 1.22ns 4.24ns 0.0299 0 0 2.22 KB
#5543 SendAsync netcoreapp3.1 5.04μs 2.29ns 8.86ns 0.0377 0 0 2.76 KB
#5543 SendAsync net472 7.7μs 2.72ns 10.5ns 0.497 0 0 3.15 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.48μs 1.11ns 4.3ns 0.0226 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 2.34μs 1.33ns 4.97ns 0.0223 0 0 1.64 KB
master EnrichedLog net472 2.74μs 0.723ns 2.71ns 0.249 0 0 1.57 KB
#5543 EnrichedLog net6.0 1.53μs 0.981ns 3.67ns 0.0229 0 0 1.64 KB
#5543 EnrichedLog netcoreapp3.1 2.18μs 0.789ns 2.95ns 0.022 0 0 1.64 KB
#5543 EnrichedLog net472 2.54μs 1.71ns 6.61ns 0.25 0 0 1.57 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 117μs 235ns 912ns 0.0581 0 0 4.28 KB
master EnrichedLog netcoreapp3.1 120μs 173ns 669ns 0 0 0 4.28 KB
master EnrichedLog net472 150μs 126ns 471ns 0.678 0.226 0 4.46 KB
#5543 EnrichedLog net6.0 114μs 184ns 711ns 0.0568 0 0 4.28 KB
#5543 EnrichedLog netcoreapp3.1 119μs 239ns 925ns 0 0 0 4.28 KB
#5543 EnrichedLog net472 150μs 111ns 430ns 0.673 0.224 0 4.46 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.04μs 1.88ns 7.29ns 0.0304 0 0 2.2 KB
master EnrichedLog netcoreapp3.1 4.12μs 1.73ns 6.68ns 0.0296 0 0 2.2 KB
master EnrichedLog net472 4.98μs 1.46ns 5.66ns 0.32 0 0 2.02 KB
#5543 EnrichedLog net6.0 3.1μs 2.1ns 8.12ns 0.0309 0 0 2.2 KB
#5543 EnrichedLog netcoreapp3.1 4.35μs 1.55ns 6.01ns 0.0284 0 0 2.2 KB
#5543 EnrichedLog net472 4.92μs 1.66ns 6.45ns 0.319 0 0 2.02 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.35μs 0.629ns 2.44ns 0.0161 0 0 1.14 KB
master SendReceive netcoreapp3.1 1.81μs 1.65ns 6.38ns 0.0154 0 0 1.14 KB
master SendReceive net472 2.18μs 1.14ns 4.4ns 0.183 0.00109 0 1.16 KB
#5543 SendReceive net6.0 1.34μs 1.05ns 4.08ns 0.0161 0 0 1.14 KB
#5543 SendReceive netcoreapp3.1 1.85μs 0.869ns 3.25ns 0.0158 0 0 1.14 KB
#5543 SendReceive net472 2.09μs 1.06ns 4.09ns 0.183 0.00104 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.87μs 0.871ns 3.26ns 0.0215 0 0 1.6 KB
master EnrichedLog netcoreapp3.1 3.83μs 3.39ns 13.1ns 0.0229 0 0 1.65 KB
master EnrichedLog net472 4.42μs 1.99ns 7.44ns 0.323 0 0 2.04 KB
#5543 EnrichedLog net6.0 2.79μs 2.44ns 9.46ns 0.0223 0 0 1.6 KB
#5543 EnrichedLog netcoreapp3.1 3.92μs 1.63ns 5.89ns 0.0215 0 0 1.65 KB
#5543 EnrichedLog net472 4.35μs 2.97ns 11.5ns 0.323 0 0 2.04 KB
Benchmarks.Trace.SpanBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 409ns 0.145ns 0.561ns 0.00811 0 0 576 B
master StartFinishSpan netcoreapp3.1 589ns 0.232ns 0.867ns 0.00767 0 0 576 B
master StartFinishSpan net472 627ns 0.305ns 1.18ns 0.0915 0 0 578 B
master StartFinishScope net6.0 472ns 0.106ns 0.411ns 0.0099 0 0 696 B
master StartFinishScope netcoreapp3.1 740ns 0.351ns 1.36ns 0.00934 0 0 696 B
master StartFinishScope net472 862ns 0.449ns 1.74ns 0.104 0 0 658 B
#5543 StartFinishSpan net6.0 397ns 0.212ns 0.735ns 0.00798 0 0 576 B
#5543 StartFinishSpan netcoreapp3.1 579ns 0.412ns 1.54ns 0.00779 0 0 576 B
#5543 StartFinishSpan net472 636ns 0.339ns 1.27ns 0.0915 0 0 578 B
#5543 StartFinishScope net6.0 514ns 0.367ns 1.42ns 0.0097 0 0 696 B
#5543 StartFinishScope netcoreapp3.1 706ns 0.412ns 1.6ns 0.0092 0 0 696 B
#5543 StartFinishScope net472 839ns 0.557ns 2.08ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Slower :warning: Same allocations :heavy_check_mark:

Slower :warning: in #5543

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0 1.142 672.83 768.08

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 673ns 0.338ns 1.31ns 0.00975 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 911ns 0.42ns 1.63ns 0.00915 0 0 696 B
master RunOnMethodBegin net472 1.09μs 0.277ns 1.04ns 0.104 0 0 658 B
#5543 RunOnMethodBegin net6.0 768ns 0.531ns 1.99ns 0.00956 0 0 696 B
#5543 RunOnMethodBegin netcoreapp3.1 938ns 0.372ns 1.39ns 0.00907 0 0 696 B
#5543 RunOnMethodBegin net472 1.1μs 0.347ns 1.35ns 0.105 0 0 658 B

andrewlock avatar May 08 '24 17:05 andrewlock

Throughput/Crank Report :zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5543) (11.791M)   : 0, 11790915
    master (11.650M)   : 0, 11650062
    benchmarks/2.9.0 (11.513M)   : 0, 11512780

    section Automatic
    This PR (5543) (8.013M)   : 0, 8013496
    master (7.797M)   : 0, 7797173
    benchmarks/2.9.0 (8.207M)   : 0, 8206882

    section Trace stats
    master (7.953M)   : 0, 7952557

    section Manual
    master (11.435M)   : 0, 11434662

    section Manual + Automatic
    This PR (5543) (7.428M)   : 0, 7428378
    master (7.236M)   : 0, 7235545

    section DD_TRACE_ENABLED=0
    master (10.644M)   : 0, 10643879


gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5543) (9.699M)   : 0, 9698746
    benchmarks/2.9.0 (9.495M)   : 0, 9494663

    section Automatic
    This PR (5543) (6.594M)   : 0, 6594222

    section Manual + Automatic
    This PR (5543) (6.161M)   : 0, 6160784


gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5543) (10.081M)   : 0, 10080955
    master (10.272M)   : 0, 10272134

    section Automatic
    This PR (5543) (6.983M)   : 0, 6982743
    master (6.659M)   : 0, 6658808

    section Trace stats
    master (7.368M)   : 0, 7367546

    section Manual
    master (10.042M)   : 0, 10041533

    section Manual + Automatic
    This PR (5543) (6.423M)   : 0, 6422932
    master (6.184M)   : 0, 6183891

    section DD_TRACE_ENABLED=0
    master (9.352M)   : 0, 9352368

andrewlock avatar May 08 '24 17:05 andrewlock

Datadog Report

Branch report: dudik/el-nullable-type Commit report: 28f8595 Test service: dd-trace-dotnet

:x: 35 Failed (0 Known Flaky), 337068 Passed, 2032 Skipped, 21h 20m 44.8s Total Time

:x: Failed Tests (35)

This report shows up to 5 failed tests.

  • Profiler_allocation - scenarios - Details

    Expand for error

  • Profiler_allocation - scenarios - Details

    Expand for error

  • Profiler_allocation - scenarios - Details

    Expand for error

  • Profiler_allocation_cpu_walltime - scenarios - Details

    Expand for error

  • Profiler_allocation_cpu_walltime - scenarios - Details

    Expand for error

datadog-ddstaging[bot] avatar May 28 '24 15:05 datadog-ddstaging[bot]

Datadog Report

Branch report: dudik/el-nullable-type Commit report: bda27df Test service: dd-trace-dotnet

:white_check_mark: 0 Failed, 354262 Passed, 2257 Skipped, 23h 18m 37.62s Total Time :hourglass: 1 Performance Regression

:hourglass: Performance Regressions vs Default Branch (1)

  • Profiler_contention - scenarios 2.35s (+77.1ms, +3%) - Details

datadog-ddstaging[bot] avatar Jul 23 '24 11:07 datadog-ddstaging[bot]