dd-trace-dotnet icon indicating copy to clipboard operation
dd-trace-dotnet copied to clipboard

Published 20 hours ago • verified publisher icon DataDog

[ASM] don't run ContextTests as it generates crazy amounts of logs

Open robertpi opened this issue 1 year ago • 5 comments

Summary of changes

Stop generating logs that clog up CI.

robertpi avatar Feb 22 '24 16:02 robertpi

Datadog Report

Branch report: robert/asm/dont-long-crazy-amounts-of-warnings-in-unit-tests Commit report: 2dd91d6 Test service: dd-trace-dotnet

:x: 1 Failed (0 Known Flaky), 311937 Passed, 1629 Skipped, 46m 50.86s Wall Time :hourglass: 1 Performance Regression

:x: Failed Tests (1)

  • NoExceptions - Datadog.Trace.ClrProfiler.IntegrationTests.SmokeTests.EnumerateAssemblyReferencesTest - Details

    Expand for error 
    xpected exit code: 0, actual exit code: -1073740791.

:hourglass: Performance Regressions vs Default Branch (1)

  • Profiler_liveheap_cpu_walltime - scenarios 4.63s (+209.25ms, +5%) - Details

datadog-ddstaging[bot] avatar Feb 22 '24 16:02 datadog-ddstaging[bot]

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5227) - mean (76ms)  : 65, 87
     .   : milestone, 76,
    master - mean (74ms)  : 66, 82
     .   : milestone, 74,

    section CallTarget+Inlining+NGEN
    This PR (5227) - mean (986ms)  : 961, 1011
     .   : milestone, 986,
    master - mean (982ms)  : 956, 1008
     .   : milestone, 982,


gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5227) - mean (111ms)  : 107, 115
     .   : milestone, 111,
    master - mean (111ms)  : 108, 113
     .   : milestone, 111,

    section CallTarget+Inlining+NGEN
    This PR (5227) - mean (717ms)  : 693, 741
     .   : milestone, 717,
    master - mean (712ms)  : 692, 732
     .   : milestone, 712,


gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5227) - mean (97ms)  : 94, 100
     .   : milestone, 97,
    master - mean (94ms)  : 91, 96
     .   : milestone, 94,

    section CallTarget+Inlining+NGEN
    This PR (5227) - mean (669ms)  : 645, 692
     .   : milestone, 669,
    master - mean (667ms)  : 635, 698
     .   : milestone, 667,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5227) - mean (188ms)  : 184, 191
     .   : milestone, 188,
    master - mean (187ms)  : 182, 192
     .   : milestone, 187,

    section CallTarget+Inlining+NGEN
    This PR (5227) - mean (1,061ms)  : 1037, 1085
     .   : milestone, 1061,
    master - mean (1,054ms)  : 1034, 1074
     .   : milestone, 1054,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5227) - mean (272ms)  : 265, 278
     .   : milestone, 272,
    master - mean (269ms)  : 265, 274
     .   : milestone, 269,

    section CallTarget+Inlining+NGEN
    This PR (5227) - mean (1,060ms)  : 1029, 1091
     .   : milestone, 1060,
    master - mean (1,056ms)  : 1027, 1085
     .   : milestone, 1056,


gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5227) - mean (259ms)  : 255, 263
     .   : milestone, 259,
    master - mean (258ms)  : 254, 263
     .   : milestone, 258,

    section CallTarget+Inlining+NGEN
    This PR (5227) - mean (993ms)  : 964, 1022
     .   : milestone, 993,
    master - mean (995ms)  : 969, 1021
     .   : milestone, 995,

andrewlock avatar Feb 22 '24 16:02 andrewlock

Benchmarks Report :snail:

Benchmarks for #5227 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.350
  • 3 benchmarks are slower, with geometric mean 1.155
  • 2 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.76μs 44.8ns 233ns 0.0252 0.0126 0 7.49 KB
master StartStopWithChild netcoreapp3.1 10.4μs 57.4ns 344ns 0.0259 0.0104 0 7.59 KB
master StartStopWithChild net472 16.8μs 64.3ns 241ns 1.34 0.361 0.101 7.96 KB
#5227 StartStopWithChild net6.0 8.64μs 39.6ns 198ns 0.0205 0.00822 0 7.49 KB
#5227 StartStopWithChild netcoreapp3.1 11.4μs 63.9ns 414ns 0.0282 0.0113 0 7.59 KB
#5227 StartStopWithChild net472 17.1μs 64.3ns 249ns 1.33 0.324 0.0996 7.96 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 449μs 366ns 1.37μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 611μs 167ns 624ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 827μs 338ns 1.31μs 0.406 0 0 3.3 KB
#5227 WriteAndFlushEnrichedTraces net6.0 468μs 193ns 721ns 0 0 0 2.7 KB
#5227 WriteAndFlushEnrichedTraces netcoreapp3.1 615μs 200ns 720ns 0 0 0 2.7 KB
#5227 WriteAndFlushEnrichedTraces net472 802μs 239ns 926ns 0.403 0 0 3.3 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Slower :warning: Same allocations :heavy_check_mark:

Slower :warning: in #5227

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorMoreComplexBody‑net472 1.179 3,773.77 4,450.18
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑net6.0 1.141 138.75 158.28

Faster :tada: in #5227

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑netcoreapp3.1 1.350 274.44 203.22

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 40.7μs 31.8ns 119ns 0.0202 0 0 2.36 KB
master AllCycleSimpleBody netcoreapp3.1 44.3μs 102ns 395ns 0.0221 0 0 2.34 KB
master AllCycleSimpleBody net472 46.9μs 23.3ns 90.4ns 0.374 0 0 2.41 KB
master AllCycleMoreComplexBody net6.0 212μs 76.9ns 298ns 0.106 0 0 9.84 KB
master AllCycleMoreComplexBody netcoreapp3.1 226μs 206ns 798ns 0.113 0 0 9.73 KB
master AllCycleMoreComplexBody net472 246μs 66.2ns 256ns 1.46 0 0 9.91 KB
master ObjectExtractorSimpleBody net6.0 139ns 0.0667ns 0.25ns 0.0039 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 275ns 0.122ns 0.472ns 0.0037 0 0 272 B
master ObjectExtractorSimpleBody net472 172ns 0.0665ns 0.258ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 3μs 0.983ns 3.68ns 0.0528 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 3.99μs 1.33ns 5.17ns 0.05 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 3.77μs 1.05ns 4.06ns 0.602 0.00565 0 3.8 KB
#5227 AllCycleSimpleBody net6.0 40.8μs 14.1ns 54.4ns 0.0202 0 0 2.36 KB
#5227 AllCycleSimpleBody netcoreapp3.1 44.8μs 131ns 507ns 0.0222 0 0 2.34 KB
#5227 AllCycleSimpleBody net472 47.1μs 27.4ns 106ns 0.375 0 0 2.41 KB
#5227 AllCycleMoreComplexBody net6.0 213μs 50.9ns 197ns 0.106 0 0 9.84 KB
#5227 AllCycleMoreComplexBody netcoreapp3.1 225μs 146ns 526ns 0.112 0 0 9.73 KB
#5227 AllCycleMoreComplexBody net472 241μs 45.6ns 170ns 1.57 0 0 9.91 KB
#5227 ObjectExtractorSimpleBody net6.0 158ns 0.139ns 0.539ns 0.00394 0 0 280 B
#5227 ObjectExtractorSimpleBody netcoreapp3.1 203ns 0.282ns 1.09ns 0.00366 0 0 272 B
#5227 ObjectExtractorSimpleBody net472 171ns 0.0673ns 0.261ns 0.0446 0 0 281 B
#5227 ObjectExtractorMoreComplexBody net6.0 3.22μs 1.05ns 3.65ns 0.0536 0 0 3.78 KB
#5227 ObjectExtractorMoreComplexBody netcoreapp3.1 4.05μs 2.79ns 10.1ns 0.0505 0 0 3.69 KB
#5227 ObjectExtractorMoreComplexBody net472 4.45μs 1.45ns 5.63ns 0.601 0.00666 0 3.8 KB
Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EncodeArgs net6.0 53μs 50ns 187ns 0.421 0 0 30.94 KB
master EncodeArgs netcoreapp3.1 68.2μs 54.5ns 211ns 0.407 0 0 31.47 KB
master EncodeArgs net472 84.2μs 50.1ns 194ns 5.12 0.084 0 32.27 KB
master EncodeLegacyArgs net6.0 128μs 225ns 870ns 0.442 0 0 33.89 KB
master EncodeLegacyArgs netcoreapp3.1 157μs 680ns 2.63μs 0.414 0 0 34.09 KB
master EncodeLegacyArgs net472 216μs 141ns 528ns 5.52 0.417 0 34.99 KB
#5227 EncodeArgs net6.0 51.1μs 51.7ns 187ns 0.437 0 0 30.94 KB
#5227 EncodeArgs netcoreapp3.1 71.3μs 107ns 416ns 0.421 0 0 31.47 KB
#5227 EncodeArgs net472 84.6μs 50.5ns 196ns 5.11 0.0844 0 32.27 KB
#5227 EncodeLegacyArgs net6.0 130μs 134ns 501ns 0.441 0 0 33.89 KB
#5227 EncodeLegacyArgs netcoreapp3.1 155μs 220ns 824ns 0.471 0 0 34.09 KB
#5227 EncodeLegacyArgs net472 209μs 257ns 961ns 5.47 0.421 0 34.99 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWafRealisticBenchmark net6.0 189μs 515ns 1.99μs 0.0953 0 0 6.51 KB
master RunWafRealisticBenchmark netcoreapp3.1 205μs 296ns 1.15μs 0 0 0 6.49 KB
master RunWafRealisticBenchmark net472 224μs 110ns 424ns 1 0 0 6.59 KB
master RunWafRealisticBenchmarkWithAttack net6.0 128μs 294ns 1.14μs 0.0631 0 0 4.15 KB
master RunWafRealisticBenchmarkWithAttack netcoreapp3.1 135μs 170ns 659ns 0 0 0 4.15 KB
master RunWafRealisticBenchmarkWithAttack net472 147μs 44.5ns 173ns 0.592 0 0 4.19 KB
#5227 RunWafRealisticBenchmark net6.0 190μs 167ns 648ns 0.0944 0 0 6.51 KB
#5227 RunWafRealisticBenchmark netcoreapp3.1 203μs 402ns 1.56μs 0 0 0 6.49 KB
#5227 RunWafRealisticBenchmark net472 223μs 135ns 525ns 1 0 0 6.59 KB
#5227 RunWafRealisticBenchmarkWithAttack net6.0 125μs 76.6ns 265ns 0.0626 0 0 4.15 KB
#5227 RunWafRealisticBenchmarkWithAttack netcoreapp3.1 134μs 278ns 1.08μs 0 0 0 4.14 KB
#5227 RunWafRealisticBenchmarkWithAttack net472 147μs 73.1ns 283ns 0.659 0 0 4.19 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 172μs 153ns 593ns 0.259 0 0 18.26 KB
master SendRequest netcoreapp3.1 191μs 453ns 1.75μs 0.193 0 0 20.42 KB
master SendRequest net472 8.56E‑05ns 8.56E‑05ns 0.00032ns 0 0 0 0 b
#5227 SendRequest net6.0 173μs 74.1ns 267ns 0.258 0 0 18.26 KB
#5227 SendRequest netcoreapp3.1 194μs 219ns 849ns 0.195 0 0 20.42 KB
#5227 SendRequest net472 2.54E‑05ns 2.54E‑05ns 9.52E‑05ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 552μs 270ns 1.01μs 0.546 0 0 41.63 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 654μs 1.21μs 4.68μs 0.331 0 0 41.65 KB
master WriteAndFlushEnrichedTraces net472 824μs 3.75μs 15.9μs 8.17 2.45 0.408 53.26 KB
#5227 WriteAndFlushEnrichedTraces net6.0 560μs 1.16μs 4.48μs 0.563 0 0 41.81 KB
#5227 WriteAndFlushEnrichedTraces netcoreapp3.1 662μs 1.49μs 5.77μs 0.326 0 0 41.6 KB
#5227 WriteAndFlushEnrichedTraces net472 833μs 3.99μs 15.4μs 8.33 2.5 0.417 53.25 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.18μs 0.505ns 1.89ns 0.0105 0 0 776 B
master ExecuteNonQuery netcoreapp3.1 1.46μs 0.942ns 3.65ns 0.0102 0 0 776 B
master ExecuteNonQuery net472 1.72μs 0.529ns 1.98ns 0.117 0 0 738 B
#5227 ExecuteNonQuery net6.0 1.08μs 0.741ns 2.87ns 0.0107 0 0 776 B
#5227 ExecuteNonQuery netcoreapp3.1 1.46μs 0.939ns 3.64ns 0.0105 0 0 776 B
#5227 ExecuteNonQuery net472 1.72μs 0.937ns 3.51ns 0.117 0 0 738 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.24μs 0.655ns 2.45ns 0.013 0 0 944 B
master CallElasticsearch netcoreapp3.1 1.58μs 1.82ns 6.8ns 0.0125 0 0 944 B
master CallElasticsearch net472 2.62μs 1.53ns 5.93ns 0.153 0 0 963 B
master CallElasticsearchAsync net6.0 1.33μs 0.551ns 2.06ns 0.0128 0 0 920 B
master CallElasticsearchAsync netcoreapp3.1 1.57μs 1.26ns 4.71ns 0.0133 0 0 992 B
master CallElasticsearchAsync net472 2.51μs 0.753ns 2.92ns 0.162 0 0 1.02 KB
#5227 CallElasticsearch net6.0 1.16μs 1.09ns 4.22ns 0.0135 0 0 944 B
#5227 CallElasticsearch netcoreapp3.1 1.51μs 1.67ns 6.49ns 0.0129 0 0 944 B
#5227 CallElasticsearch net472 2.52μs 1.2ns 4.64ns 0.153 0 0 963 B
#5227 CallElasticsearchAsync net6.0 1.29μs 0.876ns 3.39ns 0.013 0 0 920 B
#5227 CallElasticsearchAsync netcoreapp3.1 1.58μs 1.09ns 4.08ns 0.0137 0 0 992 B
#5227 CallElasticsearchAsync net472 2.54μs 1.13ns 4.36ns 0.162 0 0 1.02 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.22μs 0.696ns 2.7ns 0.0129 0 0 920 B
master ExecuteAsync netcoreapp3.1 1.69μs 0.548ns 1.98ns 0.0119 0 0 920 B
master ExecuteAsync net472 1.85μs 0.328ns 1.18ns 0.14 0 0 883 B
#5227 ExecuteAsync net6.0 1.3μs 0.514ns 1.99ns 0.0124 0 0 920 B
#5227 ExecuteAsync netcoreapp3.1 1.66μs 2.07ns 7.74ns 0.0125 0 0 920 B
#5227 ExecuteAsync net472 1.83μs 0.428ns 1.66ns 0.14 0 0 883 B
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.17μs 2.54ns 9.83ns 0.0291 0 0 2.1 KB
master SendAsync netcoreapp3.1 5.09μs 10.9ns 42.3ns 0.0357 0 0 2.64 KB
master SendAsync net472 7.67μs 1.83ns 6.84ns 0.524 0 0 3.31 KB
#5227 SendAsync net6.0 4.03μs 5.57ns 21.6ns 0.0282 0 0 2.1 KB
#5227 SendAsync netcoreapp3.1 4.88μs 1.81ns 7.01ns 0.0364 0 0 2.64 KB
#5227 SendAsync net472 7.72μs 2.7ns 10.4ns 0.524 0 0 3.31 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed :heavy_check_mark: More allocations :warning:

More allocations :warning: in #5227

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472 59.83 KB 60.49 KB 656 B 1.10%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1 201.53 KB 202.8 KB 1.27 KB 0.63%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark net6.0 63μs 778ns 7.78μs 0 0 0 43.44 KB
master StringConcatBenchmark netcoreapp3.1 53.8μs 241ns 932ns 0 0 0 42.64 KB
master StringConcatBenchmark net472 37.7μs 78.9ns 295ns 0 0 0 59.83 KB
master StringConcatAspectBenchmark net6.0 258μs 3.72μs 35.3μs 0 0 0 204.89 KB
master StringConcatAspectBenchmark netcoreapp3.1 275μs 1.27μs 4.9μs 0 0 0 201.53 KB
master StringConcatAspectBenchmark net472 234μs 2.65μs 24.6μs 0 0 0 221.18 KB
#5227 StringConcatBenchmark net6.0 61.5μs 714ns 7.07μs 0 0 0 43.44 KB
#5227 StringConcatBenchmark netcoreapp3.1 60.3μs 830ns 8.13μs 0 0 0 42.64 KB
#5227 StringConcatBenchmark net472 37.3μs 125ns 432ns 0 0 0 60.49 KB
#5227 StringConcatAspectBenchmark net6.0 266μs 5.11μs 50.3μs 0 0 0 204.42 KB
#5227 StringConcatAspectBenchmark netcoreapp3.1 283μs 1.53μs 7.93μs 0 0 0 202.8 KB
#5227 StringConcatAspectBenchmark net472 226μs 2.11μs 19.9μs 0 0 0 221.18 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.41μs 0.56ns 2.1ns 0.0221 0 0 1.58 KB
master EnrichedLog netcoreapp3.1 2.28μs 0.704ns 2.63ns 0.0218 0 0 1.58 KB
master EnrichedLog net472 2.58μs 1.49ns 5.38ns 0.239 0 0 1.51 KB
#5227 EnrichedLog net6.0 1.56μs 0.783ns 3.03ns 0.022 0 0 1.58 KB
#5227 EnrichedLog netcoreapp3.1 2.17μs 0.848ns 3.06ns 0.0209 0 0 1.58 KB
#5227 EnrichedLog net472 2.59μs 2.23ns 8.63ns 0.239 0 0 1.51 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 113μs 146ns 564ns 0.0561 0 0 4.22 KB
master EnrichedLog netcoreapp3.1 117μs 97.1ns 350ns 0 0 0 4.22 KB
master EnrichedLog net472 150μs 110ns 426ns 0.675 0.225 0 4.4 KB
#5227 EnrichedLog net6.0 113μs 90.7ns 339ns 0.0569 0 0 4.22 KB
#5227 EnrichedLog netcoreapp3.1 118μs 122ns 472ns 0 0 0 4.22 KB
#5227 EnrichedLog net472 147μs 56ns 209ns 0.663 0.221 0 4.4 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.17μs 0.832ns 3ns 0.03 0 0 2.14 KB
master EnrichedLog netcoreapp3.1 4.28μs 2.07ns 8.01ns 0.028 0 0 2.14 KB
master EnrichedLog net472 4.96μs 1.17ns 4.52ns 0.308 0 0 1.95 KB
#5227 EnrichedLog net6.0 3.17μs 1.04ns 4.01ns 0.0295 0 0 2.14 KB
#5227 EnrichedLog netcoreapp3.1 4.26μs 1.81ns 6.52ns 0.0298 0 0 2.14 KB
#5227 EnrichedLog net472 4.79μs 2.23ns 8.64ns 0.309 0 0 1.95 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.37μs 0.808ns 3.02ns 0.0151 0 0 1.11 KB
master SendReceive netcoreapp3.1 1.71μs 0.494ns 1.78ns 0.0153 0 0 1.11 KB
master SendReceive net472 2.1μs 1.86ns 7.21ns 0.178 0 0 1.12 KB
#5227 SendReceive net6.0 1.35μs 0.537ns 2.08ns 0.0156 0 0 1.11 KB
#5227 SendReceive netcoreapp3.1 1.85μs 0.906ns 3.51ns 0.0148 0 0 1.11 KB
#5227 SendReceive net472 2.19μs 1.78ns 6.65ns 0.178 0 0 1.12 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.74μs 0.468ns 1.75ns 0.0206 0 0 1.54 KB
master EnrichedLog netcoreapp3.1 3.87μs 0.824ns 2.97ns 0.0215 0 0 1.58 KB
master EnrichedLog net472 4.35μs 1.52ns 5.7ns 0.313 0 0 1.97 KB
#5227 EnrichedLog net6.0 2.73μs 1.23ns 4.77ns 0.0217 0 0 1.54 KB
#5227 EnrichedLog netcoreapp3.1 3.98μs 1.03ns 3.98ns 0.0218 0 0 1.58 KB
#5227 EnrichedLog net472 4.21μs 1.94ns 6.99ns 0.313 0 0 1.97 KB
Benchmarks.Trace.SpanBenchmark - Slower :warning: Same allocations :heavy_check_mark:

Slower :warning: in #5227

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1 1.146 716.01 820.34

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 478ns 1ns 3.88ns 0.00764 0 0 544 B
master StartFinishSpan netcoreapp3.1 715ns 1.75ns 6.77ns 0.00729 0 0 544 B
master StartFinishSpan net472 717ns 2ns 7.76ns 0.0866 0 0 546 B
master StartFinishScope net6.0 639ns 1.2ns 4.63ns 0.0092 0 0 664 B
master StartFinishScope netcoreapp3.1 841ns 1.71ns 6.63ns 0.00886 0 0 664 B
master StartFinishScope net472 963ns 3.07ns 11.9ns 0.0991 0 0 626 B
#5227 StartFinishSpan net6.0 469ns 0.201ns 0.779ns 0.00747 0 0 544 B
#5227 StartFinishSpan netcoreapp3.1 821ns 1.51ns 5.84ns 0.00735 0 0 544 B
#5227 StartFinishSpan net472 743ns 1.84ns 7.15ns 0.0864 0 0 546 B
#5227 StartFinishScope net6.0 589ns 1.08ns 4.19ns 0.0094 0 0 664 B
#5227 StartFinishScope netcoreapp3.1 885ns 1.55ns 6.01ns 0.00883 0 0 664 B
#5227 StartFinishScope net472 966ns 1.81ns 7.03ns 0.099 0 0 626 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 673ns 0.81ns 3.14ns 0.0093 0 0 664 B
master RunOnMethodBegin netcoreapp3.1 855ns 1.98ns 7.67ns 0.00913 0 0 664 B
master RunOnMethodBegin net472 1.08μs 2.05ns 7.68ns 0.0994 0 0 626 B
#5227 RunOnMethodBegin net6.0 620ns 0.83ns 3.21ns 0.0094 0 0 664 B
#5227 RunOnMethodBegin netcoreapp3.1 946ns 1.13ns 4.39ns 0.00874 0 0 664 B
#5227 RunOnMethodBegin net472 1.08μs 1.89ns 7.32ns 0.099 0 0 626 B

andrewlock avatar Feb 22 '24 18:02 andrewlock

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5227) (10.997M)   : 0, 10997281
    master (10.933M)   : 0, 10933054
    benchmarks/2.9.0 (10.728M)   : 0, 10727514

    section Automatic
    This PR (5227) (7.466M)   : 0, 7466280
    master (7.612M)   : 0, 7611980
    benchmarks/2.9.0 (7.713M)   : 0, 7712603

    section Trace stats
    This PR (5227) (7.751M)   : 0, 7751251
    master (7.956M)   : 0, 7956068

    section Manual
    This PR (5227) (9.383M)   : 0, 9382620
    master (9.707M)   : 0, 9707165

    section Manual + Automatic
    This PR (5227) (7.026M)   : 0, 7025876
    master (7.157M)   : 0, 7156780

    section Version Conflict
    This PR (5227) (6.363M)   : 0, 6363369
    master (6.507M)   : 0, 6507462


gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5227) (9.181M)   : 0, 9181462
    master (9.537M)   : 0, 9536873
    benchmarks/2.9.0 (9.658M)   : 0, 9657995

    section Automatic
    This PR (5227) (6.685M)   : 0, 6685044
    master (6.609M)   : 0, 6609253

    section Trace stats
    This PR (5227) (6.855M)   : 0, 6854506
    master (6.929M)   : 0, 6928535

    section Manual
    This PR (5227) (8.174M)   : 0, 8174182
    master (8.340M)   : 0, 8339857

    section Manual + Automatic
    This PR (5227) (6.215M)   : 0, 6214797
    master (6.215M)   : 0, 6214841

    section Version Conflict
    This PR (5227) (5.734M)   : 0, 5733571
    master (5.629M)   : 0, 5629484


gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5227) (9.873M)   : 0, 9873263
    master (10.128M)   : 0, 10128397
    benchmarks/2.9.0 (10.130M)   : 0, 10130075

    section Automatic
    This PR (5227) (7.025M)   : 0, 7024582
    master (7.013M)   : 0, 7013338
    benchmarks/2.9.0 (7.551M)   : 0, 7550738

    section Trace stats
    This PR (5227) (7.489M)   : 0, 7488529
    master (7.390M)   : 0, 7390274

    section Manual
    This PR (5227) (8.783M)   : 0, 8782978
    master (8.682M)   : 0, 8681741

    section Manual + Automatic
    This PR (5227) (6.871M)   : 0, 6871247
    master (6.778M)   : 0, 6778238

    section Version Conflict
    This PR (5227) (6.168M)   : 0, 6168312
    master (6.200M)   : 0, 6200492


gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5227) (7.309M)   : 0, 7309184
    master (7.340M)   : 0, 7340033
    benchmarks/2.9.0 (7.794M)   : 0, 7794444

    section No attack
    This PR (5227) (1.830M)   : 0, 1830281
    master (1.834M)   : 0, 1833615
    benchmarks/2.9.0 (3.183M)   : 0, 3183143

    section Attack
    This PR (5227) (1.460M)   : 0, 1460384
    master (1.465M)   : 0, 1465080
    benchmarks/2.9.0 (2.503M)   : 0, 2503046

    section Blocking
    This PR (5227) (3.134M)   : 0, 3133557
    master (3.103M)   : 0, 3103289

    section IAST default
    This PR (5227) (6.291M)   : 0, 6291374
    master (6.533M)   : 0, 6532728

    section IAST full
    This PR (5227) (5.495M)   : 0, 5494637
    master (5.574M)   : 0, 5574362

    section Base vuln
    This PR (5227) (0.965M)   : 0, 965092
    master (0.950M)   : 0, 950387

    section IAST vuln
    This PR (5227) (0.877M)   : 0, 876852
    master (0.890M)   : 0, 890239

andrewlock avatar Feb 22 '24 19:02 andrewlock

My concern was that we would clog up customer logs, not ours so much, I'm not sure this is a good option 🤔

  • I assume we do want to test this code, no?
  • I think we need to rate-limit the log message themselves if this test is using encode list lengths we expect to see? If so, then it seems customers would be spammed in the logs too?
  • Similarly, if we expect customers could have Dictionary<string,ArrayList>, should we add support for it explicitly? And if not, maybe we should have a cache of the types we've seen that we can't encode, as we're very likely to see a type again if we saw it in the previous request (it's practically guaranteed). At worst, we should make it a debug log, so that customers only get horribly spammed when they enable debug mode?

I agree we should reduce the logging that could occur in clients apps, this PR should address that: https://github.com/DataDog/dd-trace-dotnet/pull/5234

While that makes merging this test less urgent, I still we should consider whether this is a useful test, for a few reasons:

  • it using a unit test to simulate a load test, not sure it's the best way to do this
  • creating massive collections that then get truncated probably isn't that useful for a load test (we need to test truncation, but we have other unit tests that do that)
  • I'm not convinced the Dictionary<string,ArrayList> warning can happen in clients apps, if it does then there are simpler ways to address and test this.
  • it has a fairly length execution time, probably not the best use of CI resources

I believe the test could be made useful, but I don't think it is particular is in it's current form.

Input from @daniel-romano-DD and @anna-git would be useful.

robertpi avatar Feb 23 '24 14:02 robertpi

fixed the test in another way.

robertpi avatar May 06 '24 07:05 robertpi