dd-trace-dotnet icon indicating copy to clipboard operation
dd-trace-dotnet copied to clipboard
verified publisher icon DataDog

[Dynamic Instrumentation] DEBUG-2489 Add default 3rd party detection includes\excludes

Open dudikeleti opened this issue 1 year ago • 4 comments

Summary of changes

Add the default 3rd party detection includes\excludes

dudikeleti avatar Jun 21 '24 08:06 dudikeleti

Datadog Report

Branch report: dudik/3rdparty-includes-excludes Commit report: 58e97ca Test service: dd-trace-dotnet

:white_check_mark: 0 Failed, 344312 Passed, 2225 Skipped, 23h 24m 1.9s Total Time :hourglass: 3 Performance Regressions

:hourglass: Performance Regressions vs Default Branch (3)

  • CallTarget+Inlining+NGEN - Samples.HttpMessageHandler.windows.net60.json.scenarios 906.69ms (+21.65ms, +2%) - Details
  • CallTarget+Inlining+NGEN - Samples.FakeDbCommand.windows.net60.json.scenarios 702.46ms (+23.13ms, +3%) - Details
  • CallTarget+Inlining+NGEN - Samples.HttpMessageHandler.windows.net462.json.scenarios 1.16s (+18.44ms, +2%) - Details

datadog-ddstaging[bot] avatar Jun 21 '24 08:06 datadog-ddstaging[bot]

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5722) - mean (72ms)  : 64, 80
     .   : milestone, 72,
    master - mean (72ms)  : 62, 83
     .   : milestone, 72,

    section CallTarget+Inlining+NGEN
    This PR (5722) - mean (1,066ms)  : 1047, 1084
     .   : milestone, 1066,
    master - mean (1,041ms)  : 1025, 1056
     .   : milestone, 1041,


gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5722) - mean (109ms)  : 105, 114
     .   : milestone, 109,
    master - mean (108ms)  : 104, 111
     .   : milestone, 108,

    section CallTarget+Inlining+NGEN
    This PR (5722) - mean (751ms)  : 730, 772
     .   : milestone, 751,
    master - mean (726ms)  : 708, 743
     .   : milestone, 726,


gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5722) - mean (93ms)  : 88, 98
     .   : milestone, 93,
    master - mean (91ms)  : 89, 94
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (5722) - mean (702ms)  : 679, 726
     .   : milestone, 702,
    master - mean (678ms)  : 652, 704
     .   : milestone, 678,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5722) - mean (193ms)  : 189, 196
     .   : milestone, 193,
    master - mean (192ms)  : 189, 196
     .   : milestone, 192,

    section CallTarget+Inlining+NGEN
    This PR (5722) - mean (1,164ms)  : 1129, 1199
     .   : milestone, 1164,
    master - mean (1,144ms)  : 1112, 1175
     .   : milestone, 1144,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5722) - mean (276ms)  : 269, 282
     .   : milestone, 276,
    master - mean (277ms)  : 271, 283
     .   : milestone, 277,

    section CallTarget+Inlining+NGEN
    This PR (5722) - mean (922ms)  : 897, 948
     .   : milestone, 922,
    master - mean (913ms)  : 871, 954
     .   : milestone, 913,


gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5722) - mean (266ms)  : 262, 269
     .   : milestone, 266,
    master - mean (266ms)  : 262, 270
     .   : milestone, 266,

    section CallTarget+Inlining+NGEN
    This PR (5722) - mean (907ms)  : 884, 929
     .   : milestone, 907,
    master - mean (881ms)  : 855, 906
     .   : milestone, 881,

andrewlock avatar Jun 21 '24 08:06 andrewlock

Throughput/Crank Report :zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5722) (11.757M)   : 0, 11756820
    master (11.733M)   : 0, 11733443
    benchmarks/2.9.0 (11.335M)   : 0, 11335046

    section Automatic
    This PR (5722) (7.836M)   : 0, 7836120
    master (8.039M)   : 0, 8039013
    benchmarks/2.9.0 (8.124M)   : 0, 8123605

    section Trace stats
    master (8.357M)   : 0, 8356710

    section Manual
    master (11.564M)   : 0, 11563887

    section Manual + Automatic
    This PR (5722) (7.506M)   : 0, 7505724
    master (7.319M)   : 0, 7319131

    section DD_TRACE_ENABLED=0
    master (10.699M)   : 0, 10699364


gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5722) (9.506M)   : 0, 9506227

    section Automatic
    This PR (5722) (6.742M)   : 0, 6741810

    section Manual + Automatic
    This PR (5722) (6.226M)   : 0, 6225669


gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5722) (10.287M)   : 0, 10286901
    master (10.271M)   : 0, 10271035
    benchmarks/2.9.0 (10.092M)   : 0, 10091577

    section Automatic
    This PR (5722) (6.750M)   : 0, 6749953
    master (6.709M)   : 0, 6709173
    benchmarks/2.9.0 (7.470M)   : 0, 7469858

    section Trace stats
    master (7.349M)   : 0, 7348857

    section Manual
    master (10.149M)   : 0, 10148514

    section Manual + Automatic
    This PR (5722) (6.338M)   : 0, 6338472
    master (6.215M)   : 0, 6215101

    section DD_TRACE_ENABLED=0
    master (9.370M)   : 0, 9369808

andrewlock avatar Jun 21 '24 09:06 andrewlock

Benchmarks Report for tracer :snail:

Benchmarks for #5722 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.137
  • 1 benchmarks are slower, with geometric mean 1.128
  • 1 benchmarks have fewer allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 7.66μs 42ns 252ns 0.0149 0.00372 0 5.43 KB
master StartStopWithChild netcoreapp3.1 9.95μs 55.9ns 371ns 0.0252 0.0101 0 5.62 KB
master StartStopWithChild net472 15.8μs 39.5ns 153ns 1.01 0.309 0.095 6.07 KB
#5722 StartStopWithChild net6.0 7.85μs 43ns 285ns 0.0154 0.00771 0 5.43 KB
#5722 StartStopWithChild netcoreapp3.1 9.84μs 51.9ns 254ns 0.0197 0.00987 0 5.61 KB
#5722 StartStopWithChild net472 15.7μs 51ns 197ns 1.03 0.305 0.102 6.06 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 492μs 315ns 1.18μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 639μs 438ns 1.7μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 844μs 653ns 2.53μs 0.422 0 0 3.3 KB
#5722 WriteAndFlushEnrichedTraces net6.0 476μs 380ns 1.47μs 0 0 0 2.7 KB
#5722 WriteAndFlushEnrichedTraces netcoreapp3.1 654μs 391ns 1.52μs 0 0 0 2.7 KB
#5722 WriteAndFlushEnrichedTraces net472 847μs 808ns 3.02μs 0.422 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 190μs 836ns 3.02μs 0.18 0 0 18.45 KB
master SendRequest netcoreapp3.1 220μs 1.19μs 9.34μs 0.21 0 0 20.61 KB
master SendRequest net472 0.00232ns 0.000773ns 0.00279ns 0 0 0 0 b
#5722 SendRequest net6.0 204μs 1.33μs 13μs 0.191 0 0 18.45 KB
#5722 SendRequest netcoreapp3.1 234μs 1.75μs 17.4μs 0.251 0 0 20.61 KB
#5722 SendRequest net472 0.0014ns 0.000707ns 0.00265ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: Fewer allocations :tada:

Fewer allocations :tada: in #5722

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0 41.71 KB 41.42 KB -286 B -0.69%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 591μs 2.84μs 11.3μs 0.558 0 0 41.71 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 686μs 3.2μs 12.8μs 0.34 0 0 41.78 KB
master WriteAndFlushEnrichedTraces net472 867μs 4.24μs 18μs 8.3 2.62 0.437 53.28 KB
#5722 WriteAndFlushEnrichedTraces net6.0 557μs 1.61μs 6.25μs 0.566 0 0 41.42 KB
#5722 WriteAndFlushEnrichedTraces netcoreapp3.1 704μs 3.58μs 15.6μs 0.343 0 0 41.79 KB
#5722 WriteAndFlushEnrichedTraces net472 867μs 3.15μs 12.2μs 8.36 2.64 0.44 53.27 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.25μs 1.01ns 3.76ns 0.0144 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 1.78μs 1.37ns 5.31ns 0.0133 0 0 1.02 KB
master ExecuteNonQuery net472 1.98μs 3.68ns 14.3ns 0.156 0 0 987 B
#5722 ExecuteNonQuery net6.0 1.24μs 1.67ns 6.46ns 0.0142 0 0 1.02 KB
#5722 ExecuteNonQuery netcoreapp3.1 1.65μs 1.05ns 3.93ns 0.0132 0 0 1.02 KB
#5722 ExecuteNonQuery net472 2.06μs 1.6ns 6.21ns 0.157 0 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.18μs 0.802ns 3ns 0.0136 0 0 976 B
master CallElasticsearch netcoreapp3.1 1.54μs 1.52ns 5.69ns 0.0129 0 0 976 B
master CallElasticsearch net472 2.43μs 1.17ns 4.38ns 0.158 0.0012 0 995 B
master CallElasticsearchAsync net6.0 1.27μs 0.648ns 2.51ns 0.0133 0 0 952 B
master CallElasticsearchAsync netcoreapp3.1 1.62μs 1.07ns 4.02ns 0.0137 0 0 1.02 KB
master CallElasticsearchAsync net472 2.64μs 0.902ns 3.38ns 0.167 0.00131 0 1.05 KB
#5722 CallElasticsearch net6.0 1.28μs 0.588ns 2.28ns 0.0135 0 0 976 B
#5722 CallElasticsearch netcoreapp3.1 1.48μs 1.25ns 4.82ns 0.0126 0 0 976 B
#5722 CallElasticsearch net472 2.39μs 1.09ns 4.23ns 0.158 0 0 995 B
#5722 CallElasticsearchAsync net6.0 1.35μs 2.6ns 10.1ns 0.0129 0 0 952 B
#5722 CallElasticsearchAsync netcoreapp3.1 1.6μs 0.983ns 3.68ns 0.0137 0 0 1.02 KB
#5722 CallElasticsearchAsync net472 2.63μs 1.99ns 7.69ns 0.166 0 0 1.05 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.29μs 0.548ns 1.98ns 0.0131 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.62μs 0.627ns 2.43ns 0.013 0 0 952 B
master ExecuteAsync net472 1.75μs 1.04ns 4.02ns 0.145 0.000873 0 915 B
#5722 ExecuteAsync net6.0 1.17μs 0.49ns 1.83ns 0.0135 0 0 952 B
#5722 ExecuteAsync netcoreapp3.1 1.6μs 1.4ns 5.43ns 0.0128 0 0 952 B
#5722 ExecuteAsync net472 1.7μs 1.33ns 5.14ns 0.145 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.2μs 1.09ns 4.08ns 0.0313 0 0 2.22 KB
master SendAsync netcoreapp3.1 5.06μs 2.95ns 11.4ns 0.0354 0 0 2.76 KB
master SendAsync net472 7.84μs 2.42ns 9.36ns 0.498 0 0 3.15 KB
#5722 SendAsync net6.0 4μs 1.07ns 4.01ns 0.03 0 0 2.22 KB
#5722 SendAsync netcoreapp3.1 5.06μs 5.62ns 21.8ns 0.0379 0 0 2.76 KB
#5722 SendAsync net472 7.88μs 2.23ns 8.36ns 0.498 0 0 3.15 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.45μs 0.744ns 2.79ns 0.0226 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 2.31μs 1.12ns 4.18ns 0.0218 0 0 1.64 KB
master EnrichedLog net472 2.53μs 1.27ns 4.92ns 0.249 0 0 1.57 KB
#5722 EnrichedLog net6.0 1.5μs 0.495ns 1.85ns 0.0235 0 0 1.64 KB
#5722 EnrichedLog netcoreapp3.1 2.24μs 1.34ns 5.02ns 0.0227 0 0 1.64 KB
#5722 EnrichedLog net472 2.58μs 1.69ns 6.53ns 0.249 0 0 1.57 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 117μs 257ns 997ns 0.0584 0 0 4.28 KB
master EnrichedLog netcoreapp3.1 119μs 115ns 413ns 0 0 0 4.28 KB
master EnrichedLog net472 148μs 275ns 1.03μs 0.671 0.224 0 4.46 KB
#5722 EnrichedLog net6.0 115μs 220ns 851ns 0.0577 0 0 4.28 KB
#5722 EnrichedLog netcoreapp3.1 121μs 166ns 598ns 0 0 0 4.28 KB
#5722 EnrichedLog net472 148μs 332ns 1.29μs 0.667 0.222 0 4.46 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.11μs 1.12ns 4.18ns 0.0307 0 0 2.2 KB
master EnrichedLog netcoreapp3.1 4.24μs 1.65ns 6.39ns 0.0299 0 0 2.2 KB
master EnrichedLog net472 4.87μs 1.94ns 7.25ns 0.319 0 0 2.02 KB
#5722 EnrichedLog net6.0 3.05μs 0.996ns 3.73ns 0.0305 0 0 2.2 KB
#5722 EnrichedLog netcoreapp3.1 4.38μs 1.79ns 6.93ns 0.0303 0 0 2.2 KB
#5722 EnrichedLog net472 4.87μs 1.59ns 6.18ns 0.319 0 0 2.02 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.31μs 1.29ns 5ns 0.0158 0 0 1.14 KB
master SendReceive netcoreapp3.1 1.71μs 0.967ns 3.62ns 0.0155 0 0 1.14 KB
master SendReceive net472 2.08μs 0.918ns 3.55ns 0.183 0.00105 0 1.16 KB
#5722 SendReceive net6.0 1.4μs 0.466ns 1.81ns 0.0161 0 0 1.14 KB
#5722 SendReceive netcoreapp3.1 1.8μs 0.722ns 2.79ns 0.0153 0 0 1.14 KB
#5722 SendReceive net472 2.08μs 1.16ns 4.49ns 0.183 0.00104 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.77μs 0.909ns 3.52ns 0.0222 0 0 1.6 KB
master EnrichedLog netcoreapp3.1 3.92μs 1.1ns 4.11ns 0.0215 0 0 1.65 KB
master EnrichedLog net472 4.43μs 2.03ns 7.59ns 0.323 0 0 2.04 KB
#5722 EnrichedLog net6.0 2.69μs 0.718ns 2.78ns 0.0215 0 0 1.6 KB
#5722 EnrichedLog netcoreapp3.1 3.91μs 4.38ns 17ns 0.021 0 0 1.65 KB
#5722 EnrichedLog net472 4.31μs 3.02ns 11.7ns 0.322 0 0 2.04 KB
Benchmarks.Trace.SpanBenchmark - Slower :warning: Same allocations :heavy_check_mark:

Slower :warning: in #5722

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 1.128 395.05 445.58

Faster :tada: in #5722

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472 1.137 688.67 605.50

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 395ns 0.123ns 0.459ns 0.00816 0 0 576 B
master StartFinishSpan netcoreapp3.1 594ns 0.207ns 0.775ns 0.00782 0 0 576 B
master StartFinishSpan net472 689ns 0.371ns 1.44ns 0.0917 0 0 578 B
master StartFinishScope net6.0 495ns 0.179ns 0.694ns 0.00984 0 0 696 B
master StartFinishScope netcoreapp3.1 740ns 0.276ns 1.07ns 0.0092 0 0 696 B
master StartFinishScope net472 870ns 0.759ns 2.63ns 0.104 0 0 658 B
#5722 StartFinishSpan net6.0 446ns 0.121ns 0.469ns 0.00805 0 0 576 B
#5722 StartFinishSpan netcoreapp3.1 582ns 0.458ns 1.59ns 0.0077 0 0 576 B
#5722 StartFinishSpan net472 605ns 0.537ns 2.08ns 0.0917 0 0 578 B
#5722 StartFinishScope net6.0 544ns 0.185ns 0.715ns 0.00982 0 0 696 B
#5722 StartFinishScope netcoreapp3.1 720ns 1.41ns 5.28ns 0.00922 0 0 696 B
#5722 StartFinishScope net472 873ns 0.808ns 3.13ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 611ns 0.338ns 1.31ns 0.00981 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 984ns 0.438ns 1.7ns 0.00933 0 0 696 B
master RunOnMethodBegin net472 1.1μs 0.521ns 2.02ns 0.104 0 0 658 B
#5722 RunOnMethodBegin net6.0 600ns 0.265ns 1.03ns 0.00974 0 0 696 B
#5722 RunOnMethodBegin netcoreapp3.1 943ns 0.979ns 3.66ns 0.00943 0 0 696 B
#5722 RunOnMethodBegin net472 1.13μs 0.217ns 0.84ns 0.104 0 0 658 B

andrewlock avatar Jun 21 '24 09:06 andrewlock