dd-trace-dotnet icon indicating copy to clipboard operation
dd-trace-dotnet copied to clipboard
verified publisher icon DataDog

add new integrations system tests scenario

Open wconti27 opened this issue 1 year ago • 5 comments

Summary of changes

add new scenario

Reason for change

Implementation details

Test coverage

Other details

wconti27 avatar Oct 22 '24 14:10 wconti27

Datadog Report

Branch report: conti/add-new-system-tests-scenario Commit report: 55220fb Test service: dd-trace-dotnet

:white_check_mark: 0 Failed, 364030 Passed, 2076 Skipped, 15h 5m 10.68s Total Time :hourglass: 1 Performance Regression

:hourglass: Performance Regressions vs Default Branch (1)

  • Baseline_contention - scenarios 1.02s (+9.65ms, +1%) - Details

datadog-ddstaging[bot] avatar Oct 22 '24 15:10 datadog-ddstaging[bot]

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6177) - mean (70ms)  : 68, 73
     .   : milestone, 70,
    master - mean (70ms)  : 68, 72
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (6177) - mean (1,118ms)  : 1093, 1144
     .   : milestone, 1118,
    master - mean (1,111ms)  : 1094, 1128
     .   : milestone, 1111,


gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6177) - mean (111ms)  : 106, 117
     .   : milestone, 111,
    master - mean (109ms)  : 106, 112
     .   : milestone, 109,

    section CallTarget+Inlining+NGEN
    This PR (6177) - mean (774ms)  : 758, 790
     .   : milestone, 774,
    master - mean (768ms)  : 751, 784
     .   : milestone, 768,


gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6177) - mean (93ms)  : 91, 95
     .   : milestone, 93,
    master - mean (93ms)  : 89, 96
     .   : milestone, 93,

    section CallTarget+Inlining+NGEN
    This PR (6177) - mean (731ms)  : 717, 745
     .   : milestone, 731,
    master - mean (730ms)  : 711, 750
     .   : milestone, 730,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6177) - mean (191ms)  : 187, 195
     .   : milestone, 191,
    master - mean (191ms)  : 186, 195
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (6177) - mean (1,205ms)  : 1181, 1229
     .   : milestone, 1205,
    master - mean (1,204ms)  : 1176, 1232
     .   : milestone, 1204,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6177) - mean (276ms)  : 272, 280
     .   : milestone, 276,
    master - mean (275ms)  : 271, 280
     .   : milestone, 275,

    section CallTarget+Inlining+NGEN
    This PR (6177) - mean (944ms)  : 920, 968
     .   : milestone, 944,
    master - mean (950ms)  : 935, 966
     .   : milestone, 950,


gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (6177) - mean (264ms)  : 259, 268
     .   : milestone, 264,
    master - mean (265ms)  : 261, 268
     .   : milestone, 265,

    section CallTarget+Inlining+NGEN
    This PR (6177) - mean (926ms)  : 904, 947
     .   : milestone, 926,
    master - mean (928ms)  : 906, 950
     .   : milestone, 928,

andrewlock avatar Oct 22 '24 15:10 andrewlock

Throughput/Crank Report :zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6177) (11.128M)   : 0, 11127815
    master (11.138M)   : 0, 11137998
    benchmarks/2.9.0 (11.081M)   : 0, 11080577

    section Automatic
    This PR (6177) (7.406M)   : 0, 7405517
    master (7.352M)   : 0, 7351943
    benchmarks/2.9.0 (7.732M)   : 0, 7732233

    section Trace stats
    master (7.584M)   : 0, 7583665

    section Manual
    master (11.120M)   : 0, 11119525

    section Manual + Automatic
    This PR (6177) (6.893M)   : 0, 6893465
    master (6.775M)   : 0, 6774742

    section DD_TRACE_ENABLED=0
    master (10.256M)   : 0, 10256071


gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6177) (9.382M)   : 0, 9381961
    benchmarks/2.9.0 (9.798M)   : 0, 9798067

    section Automatic
    This PR (6177) (6.557M)   : 0, 6556500

    section Manual + Automatic
    This PR (6177) (6.004M)   : 0, 6003725


gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (6177) (9.900M)   : 0, 9899826
    master (9.991M)   : 0, 9990864
    benchmarks/2.9.0 (10.067M)   : 0, 10067315

    section Automatic
    This PR (6177) (6.604M)   : 0, 6604471
    master (6.646M)   : 0, 6645597
    benchmarks/2.9.0 (7.552M)   : 0, 7552193

    section Trace stats
    master (7.308M)   : 0, 7308234

    section Manual
    master (10.265M)   : 0, 10264639

    section Manual + Automatic
    This PR (6177) (6.027M)   : 0, 6027000
    master (6.226M)   : 0, 6226180

    section DD_TRACE_ENABLED=0
    master (9.511M)   : 0, 9511190

andrewlock avatar Oct 22 '24 18:10 andrewlock

Looks like there was some flake on an unrelated system test

FAILED tests/parametric/test_dynamic_configuration.py::TestDynamicConfigV2::test_tracing_client_tracing_tags[library_env1]

Re-running to see if this fixes it.

bouwkast avatar Oct 22 '24 18:10 bouwkast

Benchmarks Report for tracer :snail:

Benchmarks for #6177 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.144
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 7.69μs 43.4ns 288ns 0.019 0.00761 0 5.42 KB
master StartStopWithChild netcoreapp3.1 9.64μs 51.6ns 268ns 0.0192 0.00962 0 5.61 KB
master StartStopWithChild net472 16.1μs 64ns 248ns 1.02 0.297 0.0883 6.06 KB
#6177 StartStopWithChild net6.0 7.83μs 43.5ns 265ns 0.0185 0.00741 0 5.43 KB
#6177 StartStopWithChild netcoreapp3.1 9.98μs 33.9ns 117ns 0.0198 0.00991 0 5.62 KB
#6177 StartStopWithChild net472 16μs 67.6ns 262ns 1.04 0.324 0.0949 6.06 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 464μs 301ns 1.13μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 644μs 343ns 1.33μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 840μs 426ns 1.65μs 0.417 0 0 3.3 KB
#6177 WriteAndFlushEnrichedTraces net6.0 475μs 355ns 1.37μs 0 0 0 2.7 KB
#6177 WriteAndFlushEnrichedTraces netcoreapp3.1 640μs 333ns 1.29μs 0 0 0 2.7 KB
#6177 WriteAndFlushEnrichedTraces net472 843μs 525ns 2.03μs 0.419 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 206μs 1.21μs 12μs 0.218 0 0 18.45 KB
master SendRequest netcoreapp3.1 224μs 1.27μs 12.3μs 0.221 0 0 20.61 KB
master SendRequest net472 0.000448ns 0.000342ns 0.00133ns 0 0 0 0 b
#6177 SendRequest net6.0 193μs 1.05μs 6.14μs 0.198 0 0 18.45 KB
#6177 SendRequest netcoreapp3.1 239μs 1.32μs 10.9μs 0.222 0 0 20.61 KB
#6177 SendRequest net472 0.00264ns 0.00114ns 0.00426ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 600μs 3.16μs 18.7μs 0.604 0 0 41.48 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 696μs 3.27μs 16.3μs 0.342 0 0 41.89 KB
master WriteAndFlushEnrichedTraces net472 847μs 3.89μs 15.1μs 8.33 2.5 0.417 53.29 KB
#6177 WriteAndFlushEnrichedTraces net6.0 607μs 3.37μs 21.6μs 0.566 0 0 41.67 KB
#6177 WriteAndFlushEnrichedTraces netcoreapp3.1 679μs 3.66μs 19μs 0.359 0 0 41.8 KB
#6177 WriteAndFlushEnrichedTraces net472 867μs 3.41μs 13.2μs 8.19 2.59 0.431 53.32 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.3μs 1.43ns 5.56ns 0.0143 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 1.79μs 1.21ns 4.37ns 0.0134 0 0 1.02 KB
master ExecuteNonQuery net472 2.11μs 2.48ns 9.61ns 0.156 0 0 987 B
#6177 ExecuteNonQuery net6.0 1.3μs 1.65ns 6.39ns 0.0142 0 0 1.02 KB
#6177 ExecuteNonQuery netcoreapp3.1 1.77μs 1.68ns 6.28ns 0.0133 0 0 1.02 KB
#6177 ExecuteNonQuery net472 2.01μs 2.3ns 8.62ns 0.156 0 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Faster :tada: Same allocations :heavy_check_mark:

Faster :tada: in #6177

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch‑net6.0 1.144 1,305.95 1,141.21

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.31μs 0.603ns 2.33ns 0.0138 0 0 976 B
master CallElasticsearch netcoreapp3.1 1.54μs 1.09ns 3.91ns 0.0129 0 0 976 B
master CallElasticsearch net472 2.37μs 1.51ns 5.65ns 0.158 0 0 995 B
master CallElasticsearchAsync net6.0 1.37μs 0.72ns 2.49ns 0.0131 0 0 952 B
master CallElasticsearchAsync netcoreapp3.1 1.61μs 0.9ns 3.37ns 0.0138 0 0 1.02 KB
master CallElasticsearchAsync net472 2.62μs 2.41ns 9.35ns 0.166 0 0 1.05 KB
#6177 CallElasticsearch net6.0 1.14μs 0.602ns 2.25ns 0.0137 0 0 976 B
#6177 CallElasticsearch netcoreapp3.1 1.46μs 0.714ns 2.76ns 0.0132 0 0 976 B
#6177 CallElasticsearch net472 2.53μs 1.7ns 6.37ns 0.157 0 0 995 B
#6177 CallElasticsearchAsync net6.0 1.29μs 0.553ns 2.07ns 0.0135 0 0 952 B
#6177 CallElasticsearchAsync netcoreapp3.1 1.61μs 1.53ns 5.73ns 0.014 0 0 1.02 KB
#6177 CallElasticsearchAsync net472 2.7μs 2.32ns 8.97ns 0.167 0 0 1.05 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.19μs 0.822ns 3.07ns 0.0136 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.62μs 1.12ns 4.32ns 0.013 0 0 952 B
master ExecuteAsync net472 1.83μs 1.64ns 6.33ns 0.145 0 0 915 B
#6177 ExecuteAsync net6.0 1.3μs 0.969ns 3.49ns 0.013 0 0 952 B
#6177 ExecuteAsync netcoreapp3.1 1.61μs 0.938ns 3.51ns 0.0128 0 0 952 B
#6177 ExecuteAsync net472 1.73μs 2.09ns 7.83ns 0.145 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.32μs 0.984ns 3.55ns 0.0305 0 0 2.22 KB
master SendAsync netcoreapp3.1 5.17μs 2.95ns 11.4ns 0.0363 0 0 2.76 KB
master SendAsync net472 7.73μs 5.51ns 21.3ns 0.499 0 0 3.15 KB
#6177 SendAsync net6.0 4.21μs 5.58ns 21.6ns 0.0315 0 0 2.22 KB
#6177 SendAsync netcoreapp3.1 5.1μs 4.23ns 16.4ns 0.0379 0 0 2.76 KB
#6177 SendAsync net472 7.8μs 5.92ns 22.9ns 0.497 0 0 3.15 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.49μs 1.03ns 3.73ns 0.0228 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 2.19μs 1.11ns 4.31ns 0.0218 0 0 1.64 KB
master EnrichedLog net472 2.62μs 1.4ns 5.23ns 0.249 0 0 1.57 KB
#6177 EnrichedLog net6.0 1.55μs 0.881ns 3.41ns 0.0227 0 0 1.64 KB
#6177 EnrichedLog netcoreapp3.1 2.15μs 0.696ns 2.61ns 0.0213 0 0 1.64 KB
#6177 EnrichedLog net472 2.7μs 1.44ns 5ns 0.249 0 0 1.57 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 117μs 291ns 1.13μs 0 0 0 4.28 KB
master EnrichedLog netcoreapp3.1 121μs 159ns 594ns 0 0 0 4.28 KB
master EnrichedLog net472 153μs 105ns 380ns 0.692 0.231 0 4.46 KB
#6177 EnrichedLog net6.0 117μs 162ns 606ns 0.0576 0 0 4.28 KB
#6177 EnrichedLog netcoreapp3.1 120μs 266ns 1.03μs 0.0593 0 0 4.28 KB
#6177 EnrichedLog net472 152μs 108ns 420ns 0.68 0.227 0 4.46 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.94μs 0.999ns 3.74ns 0.031 0 0 2.2 KB
master EnrichedLog netcoreapp3.1 4.07μs 3.74ns 14.5ns 0.0286 0 0 2.2 KB
master EnrichedLog net472 4.83μs 2.45ns 9.47ns 0.319 0 0 2.02 KB
#6177 EnrichedLog net6.0 3.1μs 1.3ns 4.5ns 0.031 0 0 2.2 KB
#6177 EnrichedLog netcoreapp3.1 4.29μs 1.91ns 7.39ns 0.0297 0 0 2.2 KB
#6177 EnrichedLog net472 4.79μs 2.39ns 9.25ns 0.32 0 0 2.02 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.33μs 0.651ns 2.52ns 0.016 0 0 1.14 KB
master SendReceive netcoreapp3.1 1.72μs 0.39ns 1.41ns 0.0155 0 0 1.14 KB
master SendReceive net472 2.15μs 0.909ns 3.52ns 0.183 0.00107 0 1.16 KB
#6177 SendReceive net6.0 1.29μs 0.65ns 2.52ns 0.0161 0 0 1.14 KB
#6177 SendReceive netcoreapp3.1 1.8μs 0.93ns 3.6ns 0.0151 0 0 1.14 KB
#6177 SendReceive net472 2.13μs 1.9ns 7.36ns 0.183 0.00106 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.74μs 0.78ns 3.02ns 0.0233 0 0 1.6 KB
master EnrichedLog netcoreapp3.1 3.99μs 1.04ns 3.9ns 0.0216 0 0 1.65 KB
master EnrichedLog net472 4.38μs 3.77ns 14.6ns 0.322 0 0 2.04 KB
#6177 EnrichedLog net6.0 2.86μs 1.15ns 4.32ns 0.0228 0 0 1.6 KB
#6177 EnrichedLog netcoreapp3.1 3.8μs 1.13ns 4.37ns 0.021 0 0 1.65 KB
#6177 EnrichedLog net472 4.14μs 2.25ns 8.72ns 0.323 0 0 2.04 KB
Benchmarks.Trace.SpanBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 399ns 0.316ns 1.22ns 0.00816 0 0 576 B
master StartFinishSpan netcoreapp3.1 556ns 0.516ns 1.93ns 0.00778 0 0 576 B
master StartFinishSpan net472 748ns 0.205ns 0.711ns 0.0915 0 0 578 B
master StartFinishScope net6.0 522ns 0.366ns 1.42ns 0.00986 0 0 696 B
master StartFinishScope netcoreapp3.1 742ns 0.766ns 2.97ns 0.00923 0 0 696 B
master StartFinishScope net472 907ns 1.04ns 4.04ns 0.104 0 0 658 B
#6177 StartFinishSpan net6.0 397ns 0.164ns 0.635ns 0.00815 0 0 576 B
#6177 StartFinishSpan netcoreapp3.1 550ns 0.382ns 1.43ns 0.00793 0 0 576 B
#6177 StartFinishSpan net472 679ns 0.773ns 2.89ns 0.0917 0 0 578 B
#6177 StartFinishScope net6.0 484ns 0.418ns 1.62ns 0.0097 0 0 696 B
#6177 StartFinishScope netcoreapp3.1 759ns 0.696ns 2.7ns 0.00915 0 0 696 B
#6177 StartFinishScope net472 835ns 0.698ns 2.61ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 617ns 0.44ns 1.7ns 0.00959 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 916ns 0.614ns 2.38ns 0.00906 0 0 696 B
master RunOnMethodBegin net472 1.19μs 1.21ns 4.69ns 0.105 0 0 658 B
#6177 RunOnMethodBegin net6.0 574ns 0.627ns 2.43ns 0.00984 0 0 696 B
#6177 RunOnMethodBegin netcoreapp3.1 918ns 0.764ns 2.86ns 0.00914 0 0 696 B
#6177 RunOnMethodBegin net472 1.18μs 1.28ns 4.97ns 0.104 0 0 658 B

andrewlock avatar Oct 22 '24 19:10 andrewlock