dd-trace-dotnet icon indicating copy to clipboard operation
dd-trace-dotnet copied to clipboard

Published 20 hours ago • verified publisher icon DataDog

[ASM] Initial GraphQL Integration

Open robertpi opened this issue 2 years ago • 6 comments

Summary of changes

Integrate ASM into GraphQL. Supports graphql-dotnet and HotChocolate.

Implementation details

Pass the address graphql.server.all_resolvers to the WAF. For graphql-dotnet, each resolver are stored with its arguments when a resolver node is called. For HotChocolate, the AST of the request is scanned before the request execution to store each resolver with its arguments.

Because there is different implementation on how GraphQL request are analysed (one directly before the execution of all resolvers and the other one getting the node of resolver that will be executed), I used a dictionary that is storing the resolvers and the current Scope as the key.

Data of resolvers are gathered via the generated AST of the both graphql libraries.

Notes

Takes the work from a previous pull request and fixes a number of failing tests: https://github.com/DataDog/dd-trace-dotnet/pull/4038

Further Work

  • Allow the results of separate WAF runs to be concatenated.
  • Use the WAF's yet to be release ephemeral address

robertpi avatar Oct 31 '23 08:10 robertpi

Datadog Report

Branch report: robert/asm/graphql-waf-integration Commit report: 01db756

:white_check_mark: dd-trace-dotnet: 0 Failed, 0 New Flaky, 298592 Passed, 942 Skipped, 39m 13.34s Wall Time

datadog-ddstaging[bot] avatar Oct 31 '23 09:10 datadog-ddstaging[bot]

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4784) - mean (73ms)  : 65, 81
     .   : milestone, 73,
    master - mean (70ms)  : 62, 79
     .   : milestone, 70,

    section CallTarget+Inlining+NGEN
    This PR (4784) - mean (999ms)  : 982, 1016
     .   : milestone, 999,
    master - mean (996ms)  : 974, 1017
     .   : milestone, 996,


gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4784) - mean (106ms)  : 102, 110
     .   : milestone, 106,
    master - mean (106ms)  : 103, 109
     .   : milestone, 106,

    section CallTarget+Inlining+NGEN
    This PR (4784) - mean (685ms)  : 672, 699
     .   : milestone, 685,
    master - mean (691ms)  : 674, 708
     .   : milestone, 691,


gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4784) - mean (90ms)  : 88, 93
     .   : milestone, 90,
    master - mean (91ms)  : 87, 95
     .   : milestone, 91,

    section CallTarget+Inlining+NGEN
    This PR (4784) - mean (658ms)  : 636, 680
     .   : milestone, 658,
    master - mean (657ms)  : 637, 676
     .   : milestone, 657,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4784) - mean (189ms)  : 186, 192
     .   : milestone, 189,
    master - mean (189ms)  : 185, 193
     .   : milestone, 189,

    section CallTarget+Inlining+NGEN
    This PR (4784) - mean (1,132ms)  : 1107, 1156
     .   : milestone, 1132,
    master - mean (1,137ms)  : 1114, 1159
     .   : milestone, 1137,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4784) - mean (272ms)  : 267, 276
     .   : milestone, 272,
    master - mean (271ms)  : 269, 274
     .   : milestone, 271,

    section CallTarget+Inlining+NGEN
    This PR (4784) - mean (1,093ms)  : 1071, 1116
     .   : milestone, 1093,
    master - mean (1,093ms)  : 1069, 1116
     .   : milestone, 1093,


gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4784) - mean (261ms)  : 256, 265
     .   : milestone, 261,
    master - mean (261ms)  : 257, 265
     .   : milestone, 261,

    section CallTarget+Inlining+NGEN
    This PR (4784) - mean (1,056ms)  : 1032, 1079
     .   : milestone, 1056,
    master - mean (1,059ms)  : 1037, 1080
     .   : milestone, 1059,

andrewlock avatar Oct 31 '23 09:10 andrewlock

Benchmarks Report :snail:

Benchmarks for #4784 compared to master:

  • 2 benchmarks are faster, with geometric mean 1.164
  • 2 benchmarks are slower, with geometric mean 1.193
  • 4 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.74μs 48.6ns 322ns 0.0219 0.00876 0 7.48 KB
master StartStopWithChild netcoreapp3.1 10.9μs 57.5ns 299ns 0.0267 0.0107 0 7.58 KB
master StartStopWithChild net472 17.4μs 40.6ns 152ns 1.33 0.376 0.123 7.96 KB
#4784 StartStopWithChild net6.0 8.84μs 43.7ns 190ns 0.0178 0.00891 0 7.49 KB
#4784 StartStopWithChild netcoreapp3.1 10.8μs 58.4ns 351ns 0.0306 0.0153 0 7.58 KB
#4784 StartStopWithChild net472 17.1μs 58.7ns 227ns 1.34 0.341 0.102 7.96 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 480μs 78.7ns 272ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 621μs 203ns 761ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 775μs 298ns 1.12μs 0.388 0 0 3.3 KB
#4784 WriteAndFlushEnrichedTraces net6.0 459μs 395ns 1.53μs 0 0 0 2.7 KB
#4784 WriteAndFlushEnrichedTraces netcoreapp3.1 627μs 385ns 1.49μs 0 0 0 2.7 KB
#4784 WriteAndFlushEnrichedTraces net472 771μs 187ns 649ns 0.383 0 0 3.3 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 43.7μs 9.76ns 37.8ns 0.0215 0 0 1.77 KB
master AllCycleSimpleBody netcoreapp3.1 46.4μs 33.5ns 130ns 0.0231 0 0 1.74 KB
master AllCycleSimpleBody net472 50.2μs 74.7ns 289ns 0.27 0 0 1.81 KB
master AllCycleMoreComplexBody net6.0 214μs 65.1ns 252ns 0.107 0 0 9.25 KB
master AllCycleMoreComplexBody netcoreapp3.1 222μs 240ns 900ns 0.111 0 0 9.14 KB
master AllCycleMoreComplexBody net472 241μs 74.1ns 277ns 1.44 0 0 9.32 KB
master ObjectExtractorSimpleBody net6.0 134ns 0.0373ns 0.14ns 0.00394 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 181ns 0.126ns 0.473ns 0.00374 0 0 272 B
master ObjectExtractorSimpleBody net472 151ns 0.123ns 0.478ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 2.82μs 1.73ns 6.47ns 0.0524 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 3.71μs 0.658ns 2.46ns 0.0501 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 3.5μs 3.01ns 11.3ns 0.602 0.00525 0 3.8 KB
#4784 AllCycleSimpleBody net6.0 43.9μs 25.4ns 98.3ns 0.0226 0 0 1.77 KB
#4784 AllCycleSimpleBody netcoreapp3.1 47.2μs 67.1ns 242ns 0 0 0 1.74 KB
#4784 AllCycleSimpleBody net472 49.5μs 87.2ns 338ns 0.269 0 0 1.81 KB
#4784 AllCycleMoreComplexBody net6.0 211μs 115ns 445ns 0.106 0 0 9.25 KB
#4784 AllCycleMoreComplexBody netcoreapp3.1 225μs 493ns 1.91μs 0.112 0 0 9.14 KB
#4784 AllCycleMoreComplexBody net472 238μs 62ns 240ns 1.42 0 0 9.32 KB
#4784 ObjectExtractorSimpleBody net6.0 130ns 0.0449ns 0.168ns 0.00395 0 0 280 B
#4784 ObjectExtractorSimpleBody netcoreapp3.1 193ns 0.0619ns 0.232ns 0.00361 0 0 272 B
#4784 ObjectExtractorSimpleBody net472 151ns 0.144ns 0.539ns 0.0446 0 0 281 B
#4784 ObjectExtractorMoreComplexBody net6.0 2.73μs 1.4ns 5.24ns 0.0533 0 0 3.78 KB
#4784 ObjectExtractorMoreComplexBody netcoreapp3.1 3.63μs 0.98ns 3.67ns 0.0504 0 0 3.69 KB
#4784 ObjectExtractorMoreComplexBody net472 3.47μs 2.43ns 9.43ns 0.603 0.00521 0 3.8 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWaf(args=NestedMap (10)) net6.0 51.1μs 186ns 722ns 0.224 0 0 16.06 KB
master RunWaf(args=NestedMap (10)) netcoreapp3.1 68.5μs 311ns 1.2μs 0.198 0 0 16.06 KB
master RunWaf(args=NestedMap (10)) net472 95.2μs 503ns 2.66μs 2.56 0.0931 0 16.14 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 115μs 52.9ns 198ns 0.286 0 0 22.41 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 138μs 282ns 1.06μs 0.279 0 0 22.36 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 169μs 699ns 2.71μs 3.59 0.163 0 22.7 KB
master RunWaf(args=NestedMap (100)) net6.0 97.6μs 419ns 1.62μs 0.426 0 0 32.76 KB
master RunWaf(args=NestedMap (100)) netcoreapp3.1 133μs 734ns 4.64μs 0.409 0 0 33.33 KB
master RunWaf(args=NestedMap (100)) net472 182μs 80.9ns 313ns 5.27 0.363 0 33.67 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 170μs 312ns 1.21μs 0.492 0 0 39.1 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 208μs 1.02μs 4.22μs 0.518 0 0 39.63 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net472 267μs 92.6ns 334ns 6.28 0.512 0 40.23 KB
master RunWaf(args=NestedMap (20)) net6.0 103μs 554ns 2.83μs 0.426 0 0 32.18 KB
master RunWaf(args=NestedMap (20)) netcoreapp3.1 127μs 706ns 4.35μs 0.399 0 0 32.3 KB
master RunWaf(args=NestedMap (20)) net472 188μs 925ns 3.81μs 5.15 0.368 0 32.63 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 173μs 98.8ns 356ns 0.487 0 0 38.53 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 207μs 1.09μs 5.35μs 0.424 0 0 38.6 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 261μs 1.28μs 5.28μs 6.12 0.51 0 39.2 KB
#4784 RunWaf(args=NestedMap (10)) net6.0 54.9μs 13.3ns 49.6ns 0.223 0 0 16.06 KB
#4784 RunWaf(args=NestedMap (10)) netcoreapp3.1 69.1μs 385ns 2.53μs 0.199 0 0 16.06 KB
#4784 RunWaf(args=NestedMap (10)) net472 98.6μs 504ns 2.25μs 2.54 0.0998 0 16.14 KB
#4784 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 120μs 154ns 557ns 0.301 0 0 22.41 KB
#4784 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 134μs 147ns 571ns 0.267 0 0 22.36 KB
#4784 RunWafWithAttack(args=Neste(...)tack) [22]) net472 171μs 654ns 2.53μs 3.55 0.165 0 22.7 KB
#4784 RunWaf(args=NestedMap (100)) net6.0 104μs 50.2ns 188ns 0.46 0 0 32.76 KB
#4784 RunWaf(args=NestedMap (100)) netcoreapp3.1 136μs 70ns 271ns 0.452 0 0 33.33 KB
#4784 RunWaf(args=NestedMap (100)) net472 185μs 898ns 3.59μs 5.32 0.361 0 33.67 KB
#4784 RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 174μs 92.7ns 359ns 0.526 0 0 39.1 KB
#4784 RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 209μs 421ns 1.58μs 0.509 0 0 39.63 KB
#4784 RunWafWithAttack(args=Neste(...)tack) [23]) net472 268μs 101ns 389ns 6.3 0.514 0 40.23 KB
#4784 RunWaf(args=NestedMap (20)) net6.0 100μs 523ns 2.87μs 0.463 0 0 32.18 KB
#4784 RunWaf(args=NestedMap (20)) netcoreapp3.1 127μs 202ns 698ns 0.378 0 0 32.3 KB
#4784 RunWaf(args=NestedMap (20)) net472 181μs 103ns 386ns 5.16 0.362 0 32.63 KB
#4784 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 174μs 976ns 6.09μs 0.486 0 0 38.53 KB
#4784 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 203μs 887ns 3.43μs 0.498 0 0 38.6 KB
#4784 RunWafWithAttack(args=Neste(...)tack) [22]) net472 261μs 1.26μs 5.04μs 6.13 0.511 0 39.2 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 174μs 148ns 556ns 0.173 0 0 18.24 KB
master SendRequest netcoreapp3.1 193μs 309ns 1.2μs 0.194 0 0 20.4 KB
master SendRequest net472 0.000675ns 0.000217ns 0.000811ns 0 0 0 0 b
#4784 SendRequest net6.0 172μs 167ns 648ns 0.256 0 0 18.24 KB
#4784 SendRequest netcoreapp3.1 188μs 114ns 442ns 0.188 0 0 20.4 KB
#4784 SendRequest net472 0.00104ns 0.000399ns 0.00155ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: More allocations :warning:

More allocations :warning: in #4784

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0 41.4 KB 41.68 KB 271 B 0.65%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 561μs 1.07μs 4.14μs 0.541 0 0 41.4 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 653μs 1.2μs 4.63μs 0.329 0 0 41.69 KB
master WriteAndFlushEnrichedTraces net472 823μs 3.17μs 12.3μs 8.12 2.44 0.406 53.24 KB
#4784 WriteAndFlushEnrichedTraces net6.0 556μs 320ns 1.15μs 0.546 0 0 41.68 KB
#4784 WriteAndFlushEnrichedTraces netcoreapp3.1 649μs 404ns 1.46μs 0.326 0 0 41.69 KB
#4784 WriteAndFlushEnrichedTraces net472 855μs 3.01μs 11.6μs 8.45 2.53 0.422 53.22 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.13μs 0.506ns 1.89ns 0.0107 0 0 768 B
master ExecuteNonQuery netcoreapp3.1 1.44μs 0.566ns 2.19ns 0.0101 0 0 768 B
master ExecuteNonQuery net472 1.71μs 0.802ns 3.11ns 0.116 0 0 730 B
#4784 ExecuteNonQuery net6.0 1.08μs 0.932ns 3.61ns 0.0105 0 0 768 B
#4784 ExecuteNonQuery netcoreapp3.1 1.51μs 0.641ns 2.4ns 0.0106 0 0 768 B
#4784 ExecuteNonQuery net472 1.8μs 0.591ns 2.05ns 0.115 0 0 730 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.29μs 0.819ns 3.17ns 0.013 0 0 936 B
master CallElasticsearch netcoreapp3.1 1.49μs 0.728ns 2.72ns 0.0129 0 0 936 B
master CallElasticsearch net472 2.49μs 1.12ns 4.33ns 0.152 0 0 955 B
master CallElasticsearchAsync net6.0 1.34μs 0.529ns 2.05ns 0.0127 0 0 912 B
master CallElasticsearchAsync netcoreapp3.1 1.63μs 0.673ns 2.61ns 0.0128 0 0 984 B
master CallElasticsearchAsync net472 2.7μs 1.09ns 4.2ns 0.16 0 0 1.01 KB
#4784 CallElasticsearch net6.0 1.23μs 0.633ns 2.37ns 0.013 0 0 936 B
#4784 CallElasticsearch netcoreapp3.1 1.46μs 1.44ns 5.59ns 0.0126 0 0 936 B
#4784 CallElasticsearch net472 2.54μs 0.979ns 3.79ns 0.151 0 0 955 B
#4784 CallElasticsearchAsync net6.0 1.28μs 0.752ns 2.81ns 0.0129 0 0 912 B
#4784 CallElasticsearchAsync netcoreapp3.1 1.61μs 1.2ns 4.64ns 0.0129 0 0 984 B
#4784 CallElasticsearchAsync net472 2.63μs 0.984ns 3.81ns 0.16 0 0 1.01 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: More allocations :warning:

More allocations :warning: in #4784

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑net472 875 B 899 B 24 B 2.74%
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑net6.0 912 B 936 B 24 B 2.63%
Benchmarks.Trace.GraphQLBenchmark.ExecuteAsync‑netcoreapp3.1 912 B 936 B 24 B 2.63%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.31μs 1.02ns 3.96ns 0.0125 0 0 912 B
master ExecuteAsync netcoreapp3.1 1.65μs 2.55ns 9.89ns 0.0116 0 0 912 B
master ExecuteAsync net472 1.8μs 0.606ns 2.27ns 0.138 0 0 875 B
#4784 ExecuteAsync net6.0 1.28μs 0.824ns 3.19ns 0.0128 0 0 936 B
#4784 ExecuteAsync netcoreapp3.1 1.78μs 0.778ns 3.01ns 0.0125 0 0 936 B
#4784 ExecuteAsync net472 1.85μs 0.549ns 2.06ns 0.142 0.000924 0 899 B
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.16μs 2.85ns 10.7ns 0.029 0 0 2.1 KB
master SendAsync netcoreapp3.1 4.91μs 1.56ns 6.05ns 0.0339 0 0 2.63 KB
master SendAsync net472 7.81μs 2.37ns 9.18ns 0.523 0 0 3.31 KB
#4784 SendAsync net6.0 4.11μs 1.9ns 7.11ns 0.0288 0 0 2.1 KB
#4784 SendAsync netcoreapp3.1 4.98μs 2.3ns 8.31ns 0.0349 0 0 2.63 KB
#4784 SendAsync net472 7.98μs 1.99ns 7.7ns 0.525 0 0 3.31 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Faster :tada: Same allocations :heavy_check_mark:

Faster :tada: in #4784

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark(parameters: System.Collections.Generic.List`1[System.String])‑net6.0 1.210 61,900.00 51,150.00 bimodal

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark(parameters=Syste(...)ring] [48]) net6.0 62.3μs 746ns 7.46μs 0 0 0 43.44 KB
master StringConcatBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 53μs 261ns 1.08μs 0 0 0 42.64 KB
master StringConcatBenchmark(parameters=Syste(...)ring] [48]) net472 37.6μs 103ns 370ns 0 0 0 59.98 KB
master StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net6.0 90.3μs 2.87μs 28.7μs 0 0 0 43.29 KB
master StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 106μs 3.74μs 37.3μs 0 0 0 42.64 KB
master StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net472 50.1μs 210ns 840ns 0 0 0 57.34 KB
#4784 StringConcatBenchmark(parameters=Syste(...)ring] [48]) net6.0 51.1μs 145ns 542ns 0 0 0 43.44 KB
#4784 StringConcatBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 53.6μs 243ns 842ns 0 0 0 42.64 KB
#4784 StringConcatBenchmark(parameters=Syste(...)ring] [48]) net472 38.3μs 138ns 518ns 0 0 0 59.7 KB
#4784 StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net6.0 65.5μs 250ns 934ns 0 0 0 43.29 KB
#4784 StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 107μs 3.53μs 35.1μs 0 0 0 42.64 KB
#4784 StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net472 48.8μs 99.6ns 630ns 0 0 0 57.34 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.62μs 1.08ns 4.18ns 0.0219 0 0 1.57 KB
master EnrichedLog netcoreapp3.1 2.2μs 1.01ns 3.91ns 0.0209 0 0 1.57 KB
master EnrichedLog net472 2.55μs 2.32ns 8.7ns 0.238 0 0 1.5 KB
#4784 EnrichedLog net6.0 1.46μs 0.959ns 3.59ns 0.0221 0 0 1.57 KB
#4784 EnrichedLog netcoreapp3.1 2.19μs 5.02ns 19.4ns 0.0205 0 0 1.57 KB
#4784 EnrichedLog net472 2.55μs 2.84ns 11ns 0.237 0 0 1.5 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 113μs 203ns 787ns 0.0567 0 0 4.21 KB
master EnrichedLog netcoreapp3.1 118μs 224ns 869ns 0.0589 0 0 4.21 KB
master EnrichedLog net472 149μs 107ns 413ns 0.67 0.223 0 4.39 KB
#4784 EnrichedLog net6.0 112μs 91.3ns 354ns 0.056 0 0 4.21 KB
#4784 EnrichedLog netcoreapp3.1 116μs 100ns 361ns 0.0589 0 0 4.21 KB
#4784 EnrichedLog net472 147μs 42.6ns 159ns 0.661 0.22 0 4.39 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.89μs 0.841ns 3.26ns 0.0289 0 0 2.13 KB
master EnrichedLog netcoreapp3.1 4.38μs 2.83ns 10.9ns 0.029 0 0 2.13 KB
master EnrichedLog net472 4.93μs 4.23ns 16.4ns 0.307 0 0 1.95 KB
#4784 EnrichedLog net6.0 3.02μs 0.679ns 2.63ns 0.0306 0 0 2.13 KB
#4784 EnrichedLog netcoreapp3.1 4.15μs 1.67ns 6.47ns 0.0285 0 0 2.13 KB
#4784 EnrichedLog net472 4.96μs 2.56ns 9.9ns 0.309 0 0 1.95 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.3μs 0.998ns 3.6ns 0.0155 0 0 1.1 KB
master SendReceive netcoreapp3.1 1.72μs 0.663ns 2.48ns 0.0145 0 0 1.1 KB
master SendReceive net472 2.14μs 1.99ns 7.7ns 0.177 0 0 1.12 KB
#4784 SendReceive net6.0 1.34μs 3.13ns 12.1ns 0.0155 0 0 1.1 KB
#4784 SendReceive netcoreapp3.1 1.69μs 1.53ns 5.73ns 0.0144 0 0 1.1 KB
#4784 SendReceive net472 2.2μs 1.32ns 5.11ns 0.177 0 0 1.12 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.71μs 0.524ns 1.89ns 0.0217 0 0 1.53 KB
master EnrichedLog netcoreapp3.1 3.87μs 0.762ns 2.85ns 0.0212 0 0 1.58 KB
master EnrichedLog net472 4.5μs 2ns 7.74ns 0.311 0 0 1.97 KB
#4784 EnrichedLog net6.0 2.62μs 0.956ns 3.7ns 0.0209 0 0 1.53 KB
#4784 EnrichedLog netcoreapp3.1 3.95μs 1.79ns 6.45ns 0.0219 0 0 1.58 KB
#4784 EnrichedLog net472 4.32μs 1.47ns 5.69ns 0.31 0 0 1.97 KB
Benchmarks.Trace.SpanBenchmark - Slower :warning: Same allocations :heavy_check_mark:

Slower :warning: in #4784

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0 1.226 546.16 669.34
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 1.162 507.94 590.22

Faster :tada: in #4784

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑netcoreapp3.1 1.120 894.00 798.47

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 508ns 0.162ns 0.627ns 0.00736 0 0 536 B
master StartFinishSpan netcoreapp3.1 687ns 0.192ns 0.691ns 0.00736 0 0 536 B
master StartFinishSpan net472 761ns 0.589ns 2.28ns 0.0853 0 0 538 B
master StartFinishScope net6.0 546ns 0.195ns 0.755ns 0.00929 0 0 656 B
master StartFinishScope netcoreapp3.1 894ns 0.287ns 1.11ns 0.00886 0 0 656 B
master StartFinishScope net472 974ns 0.585ns 2.26ns 0.098 0 0 618 B
#4784 StartFinishSpan net6.0 590ns 0.206ns 0.799ns 0.00737 0 0 536 B
#4784 StartFinishSpan netcoreapp3.1 634ns 0.334ns 1.29ns 0.00737 0 0 536 B
#4784 StartFinishSpan net472 774ns 0.198ns 0.741ns 0.0852 0 0 538 B
#4784 StartFinishScope net6.0 670ns 0.431ns 1.67ns 0.00906 0 0 656 B
#4784 StartFinishScope netcoreapp3.1 799ns 0.367ns 1.42ns 0.00881 0 0 656 B
#4784 StartFinishScope net472 953ns 1.23ns 4.77ns 0.0981 0 0 618 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 710ns 0.26ns 0.972ns 0.00917 0 0 656 B
master RunOnMethodBegin netcoreapp3.1 960ns 0.311ns 1.21ns 0.00899 0 0 656 B
master RunOnMethodBegin net472 1.04μs 0.519ns 2.01ns 0.098 0 0 618 B
#4784 RunOnMethodBegin net6.0 651ns 0.169ns 0.631ns 0.00905 0 0 656 B
#4784 RunOnMethodBegin netcoreapp3.1 984ns 0.261ns 0.978ns 0.00897 0 0 656 B
#4784 RunOnMethodBegin net472 1.13μs 0.285ns 1.06ns 0.0977 0 0 618 B

andrewlock avatar Oct 31 '23 09:10 andrewlock

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4784) (11.545M)   : 0, 11544962
    master (11.398M)   : 0, 11397842
    benchmarks/2.9.0 (11.178M)   : 0, 11177752

    section Automatic
    This PR (4784) (8.012M)   : 0, 8011612
    master (7.741M)   : 0, 7741213
    benchmarks/2.9.0 (8.258M)   : 0, 8258139

    section Trace stats
    This PR (4784) (8.334M)   : 0, 8333627
    master (7.838M)   : 0, 7838308

    section Manual
    This PR (4784) (10.077M)   : 0, 10077499
    master (9.640M)   : 0, 9639565

    section Manual + Automatic
    This PR (4784) (7.516M)   : 0, 7515960
    master (7.091M)   : 0, 7090844

    section Version Conflict
    This PR (4784) (6.780M)   : 0, 6779886
    master (6.356M)   : 0, 6355720


gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4784) (9.442M)   : 0, 9441529
    master (9.626M)   : 0, 9625782
    benchmarks/2.9.0 (9.499M)   : 0, 9498942

    section Automatic
    This PR (4784) (6.539M)   : 0, 6538683
    master (6.688M)   : 0, 6687590

    section Trace stats
    This PR (4784) (6.958M)   : 0, 6958414
    master (6.778M)   : 0, 6778098

    section Manual
    This PR (4784) (8.152M)   : 0, 8152108
    master (8.409M)   : 0, 8408747

    section Manual + Automatic
    This PR (4784) (6.053M)   : 0, 6053196
    master (6.238M)   : 0, 6238241

    section Version Conflict
    This PR (4784) (5.693M)   : 0, 5692769
    master (5.861M)   : 0, 5860949


gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4784) (10.539M)   : 0, 10539340

    section Automatic
    This PR (4784) (7.269M)   : 0, 7269291

    section Trace stats
    This PR (4784) (7.599M)   : 0, 7598522

    section Manual
    This PR (4784) (9.307M)   : 0, 9306757

    section Manual + Automatic
    This PR (4784) (6.971M)   : 0, 6971153

    section Version Conflict
    This PR (4784) (6.391M)   : 0, 6390946


gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4784) (7.451M)   : 0, 7451435
    master (7.354M)   : 0, 7353594
    benchmarks/2.9.0 (7.850M)   : 0, 7850470

    section No attack
    This PR (4784) (1.975M)   : 0, 1975166
    master (1.955M)   : 0, 1954984
    benchmarks/2.9.0 (3.255M)   : 0, 3255134

    section Attack
    This PR (4784) (1.569M)   : 0, 1568937
    master (1.550M)   : 0, 1549695
    benchmarks/2.9.0 (2.479M)   : 0, 2478846

    section Blocking
    This PR (4784) (3.280M)   : 0, 3279894
    master (3.280M)   : 0, 3279832

    section IAST default
    This PR (4784) (6.614M)   : 0, 6614149
    master (6.465M)   : 0, 6465390

    section IAST full
    This PR (4784) (5.855M)   : 0, 5854719
    master (5.850M)   : 0, 5849810

    section Base vuln
    This PR (4784) (0.945M)   : 0, 944768
    master (0.945M)   : 0, 944933

    section IAST vuln
    This PR (4784) (0.867M)   : 0, 866506
    master (0.871M)   : 0, 871244

andrewlock avatar Oct 31 '23 09:10 andrewlock

Datadog Report

Branch report: robert/asm/graphql-waf-integration Commit report: f257171

:x: dd-trace-dotnet: 954 Failed (0 Known Flaky), 0 New Flaky, 299717 Passed, 925 Skipped, 39m 29.82s Wall Time

:x: Failed Tests (954)

This report shows up to 5 failed tests.

  • SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AWS.AwsSqsTests - Details

    Expand for error 
    xpected exit code: 0, actual exit code: 134.

  • SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AWS.AwsSqsTests - Details

    Expand for error 
    xpected exit code: 0, actual exit code: 134.

  • SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AWS.AwsSqsTests - Details

    Expand for error 
    xpected exit code: 0, actual exit code: 134.

  • SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AWS.AwsSqsTests - Details

    Expand for error 
    xpected exit code: 0, actual exit code: 134.

  • SubmitsTraces - Datadog.Trace.ClrProfiler.IntegrationTests.AWS.AwsSqsTests - Details

    Expand for error 
    xpected exit code: 0, actual exit code: 134.

datadog-ddstaging[bot] avatar Nov 14 '23 13:11 datadog-ddstaging[bot]

Datadog Report

Branch report: robert/asm/graphql-waf-integration Commit report: 77e3367

:white_check_mark: dd-trace-dotnet: 0 Failed, 0 New Flaky, 307089 Passed, 955 Skipped, 39m 35.34s Wall Time

datadog-ddstaging[bot] avatar Nov 22 '23 13:11 datadog-ddstaging[bot]

will probably take a different approach.

robertpi avatar May 06 '24 07:05 robertpi