dd-trace-dotnet icon indicating copy to clipboard operation
dd-trace-dotnet copied to clipboard

Published 20 hours ago • verified publisher icon DataDog

MVP for test containers (with aerospike)

Open kevingosse opened this issue 1 year ago • 4 comments

Summary of changes

Add testcontainers to our integration tests. Convert the Aerospike tests to use testcontainers.

Reason for change

Much easier to launch the tests locally.

Implementation details

For each service to run in docker, an instance of ContainerFixture must be implemented to explain what image to use and how to configure it. Then tests that depend on it must import the fixture.

In the xunit executor, we find all the docker fixtures needed to run the set of tests, and we start them. They're cleaned up after all tests cases have run (for future improvement, we can try to stop the docker images as soon as they're not needed anymore). Because fixtures are not singleton, the ContainersRegistry is used to store the objects across multiple fixtures.

Test coverage

Only Aerospike is converted in this PR.

kevingosse avatar Jan 08 '24 15:01 kevingosse

Datadog Report

Branch report: kevin/test-containers-mvp Commit report: 55f2ea0 Test service: dd-trace-dotnet

:white_check_mark: 0 Failed, 306545 Passed, 1528 Skipped, 1h 0m 31.38s Wall Time

datadog-ddstaging[bot] avatar Jan 08 '24 15:01 datadog-ddstaging[bot]

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5031) - mean (72ms)  : 60, 83
     .   : milestone, 72,
    master - mean (72ms)  : 59, 85
     .   : milestone, 72,

    section CallTarget+Inlining+NGEN
    This PR (5031) - mean (954ms)  : 932, 977
     .   : milestone, 954,
    master - mean (952ms)  : 925, 979
     .   : milestone, 952,


gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5031) - mean (106ms)  : 102, 110
     .   : milestone, 106,
    master - mean (106ms)  : 101, 110
     .   : milestone, 106,

    section CallTarget+Inlining+NGEN
    This PR (5031) - mean (697ms)  : 675, 720
     .   : milestone, 697,
    master - mean (690ms)  : 673, 707
     .   : milestone, 690,


gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5031) - mean (90ms)  : 85, 95
     .   : milestone, 90,
    master - mean (89ms)  : 86, 92
     .   : milestone, 89,

    section CallTarget+Inlining+NGEN
    This PR (5031) - mean (649ms)  : 628, 669
     .   : milestone, 649,
    master - mean (644ms)  : 622, 666
     .   : milestone, 644,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5031) - mean (189ms)  : 186, 191
     .   : milestone, 189,
    master - mean (188ms)  : 185, 191
     .   : milestone, 188,

    section CallTarget+Inlining+NGEN
    This PR (5031) - mean (1,071ms)  : 1056, 1086
     .   : milestone, 1071,
    master - mean (1,064ms)  : 1040, 1088
     .   : milestone, 1064,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5031) - mean (273ms)  : 269, 277
     .   : milestone, 273,
    master - mean (272ms)  : 267, 278
     .   : milestone, 272,

    section CallTarget+Inlining+NGEN
    This PR (5031) - mean (1,057ms)  : 1035, 1078
     .   : milestone, 1057,
    master - mean (1,051ms)  : 1016, 1087
     .   : milestone, 1051,


gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5031) - mean (262ms)  : 258, 266
     .   : milestone, 262,
    master - mean (261ms)  : 255, 266
     .   : milestone, 261,

    section CallTarget+Inlining+NGEN
    This PR (5031) - mean (1,018ms)  : 997, 1039
     .   : milestone, 1018,
    master - mean (1,019ms)  : 999, 1039
     .   : milestone, 1019,

andrewlock avatar Jan 08 '24 15:01 andrewlock

Benchmarks Report :snail:

Benchmarks for #5031 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.136
  • 2 benchmarks are slower, with geometric mean 1.166
  • 1 benchmarks have fewer allocations
  • 2 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.6μs 45ns 221ns 0.0219 0.00876 0 7.48 KB
master StartStopWithChild netcoreapp3.1 10.9μs 56ns 306ns 0.0266 0.0106 0 7.58 KB
master StartStopWithChild net472 17.3μs 48.5ns 188ns 1.32 0.331 0.0957 7.96 KB
#5031 StartStopWithChild net6.0 8.81μs 46.2ns 226ns 0.0355 0.0178 0 7.47 KB
#5031 StartStopWithChild netcoreapp3.1 10.7μs 57.8ns 306ns 0.031 0.0155 0 7.58 KB
#5031 StartStopWithChild net472 17.3μs 48.3ns 187ns 1.32 0.335 0.112 7.95 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 445μs 220ns 850ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 616μs 293ns 1.14μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 794μs 337ns 1.31μs 0.396 0 0 3.3 KB
#5031 WriteAndFlushEnrichedTraces net6.0 454μs 446ns 1.73μs 0 0 0 2.7 KB
#5031 WriteAndFlushEnrichedTraces netcoreapp3.1 614μs 208ns 804ns 0 0 0 2.7 KB
#5031 WriteAndFlushEnrichedTraces net472 781μs 264ns 989ns 0.391 0 0 3.3 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Faster :tada: Same allocations :heavy_check_mark:

Faster :tada: in #5031

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑netcoreapp3.1 1.136 226.45 199.32

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 38.8μs 22.5ns 87ns 0.0197 0 0 2.39 KB
master AllCycleSimpleBody netcoreapp3.1 41.3μs 42.5ns 159ns 0.0207 0 0 2.37 KB
master AllCycleSimpleBody net472 45.1μs 20.5ns 76.6ns 0.385 0 0 2.45 KB
master AllCycleMoreComplexBody net6.0 199μs 79.6ns 308ns 0.0992 0 0 9.87 KB
master AllCycleMoreComplexBody netcoreapp3.1 211μs 285ns 1.1μs 0.106 0 0 9.76 KB
master AllCycleMoreComplexBody net472 248μs 96.5ns 361ns 1.49 0 0 9.95 KB
master ObjectExtractorSimpleBody net6.0 144ns 0.112ns 0.436ns 0.00393 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 227ns 0.159ns 0.595ns 0.00368 0 0 272 B
master ObjectExtractorSimpleBody net472 165ns 0.124ns 0.462ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 3.04μs 1.68ns 6.04ns 0.0532 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 3.94μs 3.37ns 13.1ns 0.0512 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 3.75μs 2.44ns 9.43ns 0.602 0.00559 0 3.8 KB
#5031 AllCycleSimpleBody net6.0 39.6μs 47.9ns 185ns 0.0197 0 0 2.39 KB
#5031 AllCycleSimpleBody netcoreapp3.1 42.3μs 64.9ns 251ns 0.0211 0 0 2.37 KB
#5031 AllCycleSimpleBody net472 45.8μs 10ns 37.4ns 0.387 0 0 2.45 KB
#5031 AllCycleMoreComplexBody net6.0 202μs 204ns 790ns 0.101 0 0 9.87 KB
#5031 AllCycleMoreComplexBody netcoreapp3.1 213μs 324ns 1.26μs 0.105 0 0 9.76 KB
#5031 AllCycleMoreComplexBody net472 230μs 120ns 464ns 1.49 0 0 9.95 KB
#5031 ObjectExtractorSimpleBody net6.0 142ns 0.16ns 0.598ns 0.00394 0 0 280 B
#5031 ObjectExtractorSimpleBody netcoreapp3.1 200ns 0.201ns 0.78ns 0.0037 0 0 272 B
#5031 ObjectExtractorSimpleBody net472 182ns 0.253ns 0.98ns 0.0446 0 0 281 B
#5031 ObjectExtractorMoreComplexBody net6.0 3.01μs 2.48ns 9.29ns 0.0525 0 0 3.78 KB
#5031 ObjectExtractorMoreComplexBody netcoreapp3.1 4.02μs 1.45ns 5.21ns 0.0503 0 0 3.69 KB
#5031 ObjectExtractorMoreComplexBody net472 3.75μs 2.14ns 7.73ns 0.603 0.0056 0 3.8 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWaf(args=NestedMap (10)) net6.0 59μs 71.5ns 267ns 0.234 0 0 16.69 KB
master RunWaf(args=NestedMap (10)) netcoreapp3.1 70.9μs 201ns 779ns 0.216 0 0 16.68 KB
master RunWaf(args=NestedMap (10)) net472 103μs 216ns 837ns 2.65 0.0983 0 16.77 KB
master RunWafTwice(args=NestedMap (10)) net6.0 59.7μs 346ns 2.94μs 0.246 0 0 17.39 KB
master RunWafTwice(args=NestedMap (10)) netcoreapp3.1 72.5μs 386ns 1.93μs 0.228 0 0 17.38 KB
master RunWafTwice(args=NestedMap (10)) net472 109μs 498ns 1.93μs 2.76 0.11 0 17.5 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 116μs 530ns 2.05μs 0.328 0 0 23.03 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 129μs 135ns 486ns 0.257 0 0 22.98 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 166μs 414ns 1.49μs 3.67 0.16 0 23.33 KB
master RunWaf(args=NestedMap (100)) net6.0 101μs 24.9ns 93.1ns 0.452 0 0 33.38 KB
master RunWaf(args=NestedMap (100)) netcoreapp3.1 141μs 779ns 4.54μs 0.425 0 0 33.95 KB
master RunWaf(args=NestedMap (100)) net472 192μs 327ns 1.18μs 5.38 0.472 0 34.31 KB
master RunWafTwice(args=NestedMap (100)) net6.0 104μs 584ns 3.6μs 0.436 0 0 34.09 KB
master RunWafTwice(args=NestedMap (100)) netcoreapp3.1 143μs 616ns 2.31μs 0.408 0 0 34.65 KB
master RunWafTwice(args=NestedMap (100)) net472 198μs 163ns 632ns 5.5 0.393 0 35.03 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 164μs 58.5ns 211ns 0.491 0 0 39.73 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 200μs 106ns 367ns 0.503 0 0 40.26 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net472 261μs 369ns 1.43μs 6.44 0.515 0 40.87 KB
master RunWaf(args=NestedMap (20)) net6.0 105μs 27.3ns 98.5ns 0.45 0 0 32.81 KB
master RunWaf(args=NestedMap (20)) netcoreapp3.1 137μs 714ns 3.42μs 0.397 0 0 32.92 KB
master RunWaf(args=NestedMap (20)) net472 188μs 93.4ns 362ns 5.24 0.374 0 33.27 KB
master RunWafTwice(args=NestedMap (20)) net6.0 105μs 160ns 619ns 0.47 0 0 33.51 KB
master RunWafTwice(args=NestedMap (20)) netcoreapp3.1 141μs 240ns 930ns 0.423 0 0 33.62 KB
master RunWafTwice(args=NestedMap (20)) net472 195μs 115ns 399ns 5.39 0.392 0 34 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 164μs 61.7ns 239ns 0.516 0 0 39.15 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 202μs 564ns 2.18μs 0.509 0 0 39.22 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 260μs 300ns 1.16μs 6.22 0.508 0 39.83 KB
#5031 RunWaf(args=NestedMap (10)) net6.0 59.1μs 22.2ns 85.9ns 0.235 0 0 16.69 KB
#5031 RunWaf(args=NestedMap (10)) netcoreapp3.1 70.2μs 385ns 2.18μs 0.211 0 0 16.68 KB
#5031 RunWaf(args=NestedMap (10)) net472 98.4μs 110ns 426ns 2.64 0.0977 0 16.77 KB
#5031 RunWafTwice(args=NestedMap (10)) net6.0 59.3μs 18.8ns 67.6ns 0.247 0 0 17.39 KB
#5031 RunWafTwice(args=NestedMap (10)) netcoreapp3.1 74.6μs 262ns 1.01μs 0.227 0 0 17.38 KB
#5031 RunWafTwice(args=NestedMap (10)) net472 107μs 53.8ns 201ns 2.73 0.105 0 17.5 KB
#5031 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 114μs 618ns 3.38μs 0.326 0 0 23.03 KB
#5031 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 132μs 539ns 2.02μs 0.256 0 0 22.99 KB
#5031 RunWafWithAttack(args=Neste(...)tack) [22]) net472 164μs 83.2ns 322ns 3.68 0.164 0 23.33 KB
#5031 RunWaf(args=NestedMap (100)) net6.0 109μs 617ns 4.23μs 0.446 0 0 33.38 KB
#5031 RunWaf(args=NestedMap (100)) netcoreapp3.1 132μs 408ns 1.73μs 0.41 0 0 33.95 KB
#5031 RunWaf(args=NestedMap (100)) net472 197μs 724ns 2.8μs 5.42 0.451 0 34.31 KB
#5031 RunWafTwice(args=NestedMap (100)) net6.0 109μs 55.3ns 214ns 0.437 0 0 34.09 KB
#5031 RunWafTwice(args=NestedMap (100)) netcoreapp3.1 145μs 690ns 2.85μs 0.442 0 0 34.65 KB
#5031 RunWafTwice(args=NestedMap (100)) net472 196μs 144ns 556ns 5.49 0.392 0 35.04 KB
#5031 RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 166μs 59.2ns 221ns 0.567 0 0 39.73 KB
#5031 RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 202μs 1.08μs 5.63μs 0.507 0 0 40.26 KB
#5031 RunWafWithAttack(args=Neste(...)tack) [23]) net472 260μs 445ns 1.72μs 6.4 0.512 0 40.86 KB
#5031 RunWaf(args=NestedMap (20)) net6.0 101μs 525ns 2.46μs 0.446 0 0 32.81 KB
#5031 RunWaf(args=NestedMap (20)) netcoreapp3.1 138μs 666ns 2.75μs 0.413 0 0 32.92 KB
#5031 RunWaf(args=NestedMap (20)) net472 187μs 153ns 591ns 5.25 0.375 0 33.27 KB
#5031 RunWafTwice(args=NestedMap (20)) net6.0 113μs 363ns 1.41μs 0.455 0 0 33.51 KB
#5031 RunWafTwice(args=NestedMap (20)) netcoreapp3.1 139μs 647ns 2.5μs 0.422 0 0 33.62 KB
#5031 RunWafTwice(args=NestedMap (20)) net472 195μs 91.6ns 343ns 5.32 0.387 0 34 KB
#5031 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 171μs 389ns 1.51μs 0.493 0 0 39.15 KB
#5031 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 203μs 1.1μs 6.43μs 0.482 0 0 39.22 KB
#5031 RunWafWithAttack(args=Neste(...)tack) [22]) net472 257μs 221ns 858ns 6.26 0.511 0 39.83 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 173μs 130ns 485ns 0.173 0 0 18.25 KB
master SendRequest netcoreapp3.1 195μs 229ns 886ns 0.193 0 0 20.42 KB
master SendRequest net472 0.00136ns 0.000337ns 0.00131ns 0 0 0 0 b
#5031 SendRequest net6.0 170μs 116ns 449ns 0.17 0 0 18.25 KB
#5031 SendRequest netcoreapp3.1 193μs 222ns 859ns 0.192 0 0 20.41 KB
#5031 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: More allocations :warning:

More allocations :warning: in #5031

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0 41.44 KB 41.75 KB 316 B 0.76%
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 41.68 KB 41.95 KB 267 B 0.64%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 549μs 1.03μs 4.01μs 0.556 0 0 41.44 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 663μs 1.49μs 5.78μs 0.327 0 0 41.68 KB
master WriteAndFlushEnrichedTraces net472 821μs 3.7μs 14.3μs 8.31 2.37 0.396 53.25 KB
#5031 WriteAndFlushEnrichedTraces net6.0 539μs 1.94μs 7.52μs 0.532 0 0 41.75 KB
#5031 WriteAndFlushEnrichedTraces netcoreapp3.1 670μs 1.98μs 7.66μs 0.332 0 0 41.95 KB
#5031 WriteAndFlushEnrichedTraces net472 847μs 3.69μs 14.3μs 8.22 2.47 0.411 53.21 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.13μs 0.611ns 2.37ns 0.0107 0 0 768 B
master ExecuteNonQuery netcoreapp3.1 1.47μs 1.08ns 4.19ns 0.0104 0 0 768 B
master ExecuteNonQuery net472 1.79μs 1.23ns 4.76ns 0.116 0 0 730 B
#5031 ExecuteNonQuery net6.0 1.06μs 0.639ns 2.47ns 0.0106 0 0 768 B
#5031 ExecuteNonQuery netcoreapp3.1 1.43μs 0.511ns 1.91ns 0.01 0 0 768 B
#5031 ExecuteNonQuery net472 1.81μs 1.24ns 4.81ns 0.115 0 0 730 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.15μs 0.386ns 1.45ns 0.0133 0 0 936 B
master CallElasticsearch netcoreapp3.1 1.51μs 0.916ns 3.43ns 0.013 0 0 936 B
master CallElasticsearch net472 2.54μs 0.722ns 2.7ns 0.151 0 0 955 B
master CallElasticsearchAsync net6.0 1.2μs 0.62ns 2.32ns 0.0126 0 0 912 B
master CallElasticsearchAsync netcoreapp3.1 1.58μs 1.12ns 4.34ns 0.0128 0 0 984 B
master CallElasticsearchAsync net472 2.71μs 1.19ns 4.61ns 0.16 0 0 1.01 KB
#5031 CallElasticsearch net6.0 1.19μs 1.42ns 5.32ns 0.0127 0 0 936 B
#5031 CallElasticsearch netcoreapp3.1 1.48μs 0.842ns 3.26ns 0.012 0 0 936 B
#5031 CallElasticsearch net472 2.6μs 2.03ns 7.86ns 0.152 0 0 955 B
#5031 CallElasticsearchAsync net6.0 1.33μs 0.604ns 2.26ns 0.0124 0 0 912 B
#5031 CallElasticsearchAsync netcoreapp3.1 1.6μs 1.07ns 4.14ns 0.0135 0 0 984 B
#5031 CallElasticsearchAsync net472 2.66μs 0.792ns 2.86ns 0.16 0 0 1.01 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.26μs 0.814ns 3.05ns 0.0126 0 0 912 B
master ExecuteAsync netcoreapp3.1 1.57μs 0.732ns 2.84ns 0.0118 0 0 912 B
master ExecuteAsync net472 1.92μs 1.07ns 4.15ns 0.138 0 0 875 B
#5031 ExecuteAsync net6.0 1.3μs 0.666ns 2.49ns 0.013 0 0 912 B
#5031 ExecuteAsync netcoreapp3.1 1.51μs 0.46ns 1.72ns 0.0121 0 0 912 B
#5031 ExecuteAsync net472 1.82μs 1.67ns 6.48ns 0.139 0 0 875 B
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.13μs 1.29ns 5.01ns 0.0291 0 0 2.1 KB
master SendAsync netcoreapp3.1 4.95μs 2.04ns 7.64ns 0.0346 0 0 2.63 KB
master SendAsync net472 7.77μs 3.62ns 13ns 0.525 0 0 3.31 KB
#5031 SendAsync net6.0 3.96μs 1.64ns 6.35ns 0.0298 0 0 2.1 KB
#5031 SendAsync netcoreapp3.1 4.99μs 8.16ns 31.6ns 0.0348 0 0 2.63 KB
#5031 SendAsync net472 7.83μs 2.51ns 9.05ns 0.525 0 0 3.31 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Same speed :heavy_check_mark: Fewer allocations :tada:

Fewer allocations :tada: in #5031

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472 62.12 KB 59.26 KB -2.86 KB -4.61%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark net6.0 62.5μs 1.04μs 10.2μs 0 0 0 43.44 KB
master StringConcatBenchmark netcoreapp3.1 62.4μs 800ns 7.92μs 0 0 0 42.64 KB
master StringConcatBenchmark net472 37.1μs 72.1ns 260ns 0 0 0 62.12 KB
master StringConcatAspectBenchmark net6.0 284μs 3.45μs 31μs 0 0 0 204.05 KB
master StringConcatAspectBenchmark netcoreapp3.1 255μs 3.46μs 32.3μs 0 0 0 202.98 KB
master StringConcatAspectBenchmark net472 253μs 5.51μs 52.3μs 0 0 0 221.18 KB
#5031 StringConcatBenchmark net6.0 61.2μs 981ns 9.35μs 0 0 0 43.44 KB
#5031 StringConcatBenchmark netcoreapp3.1 54.3μs 184ns 665ns 0 0 0 42.64 KB
#5031 StringConcatBenchmark net472 38.1μs 176ns 725ns 0 0 0 59.26 KB
#5031 StringConcatAspectBenchmark net6.0 287μs 4.97μs 44.7μs 0 0 0 204.36 KB
#5031 StringConcatAspectBenchmark netcoreapp3.1 284μs 2.89μs 26.5μs 0 0 0 202.73 KB
#5031 StringConcatAspectBenchmark net472 255μs 6.07μs 58.2μs 0 0 0 221.18 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.43μs 0.566ns 2.12ns 0.0222 0 0 1.57 KB
master EnrichedLog netcoreapp3.1 2.15μs 1.21ns 4.67ns 0.0208 0 0 1.57 KB
master EnrichedLog net472 2.7μs 4.5ns 17.4ns 0.237 0 0 1.5 KB
#5031 EnrichedLog net6.0 1.54μs 1.07ns 3.87ns 0.0223 0 0 1.57 KB
#5031 EnrichedLog netcoreapp3.1 2.28μs 0.53ns 1.98ns 0.0217 0 0 1.57 KB
#5031 EnrichedLog net472 2.62μs 1.81ns 6.76ns 0.238 0 0 1.5 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 113μs 134ns 518ns 0.0567 0 0 4.21 KB
master EnrichedLog netcoreapp3.1 117μs 124ns 481ns 0.0584 0 0 4.21 KB
master EnrichedLog net472 148μs 56.1ns 217ns 0.664 0.221 0 4.39 KB
#5031 EnrichedLog net6.0 112μs 87.1ns 326ns 0.0563 0 0 4.21 KB
#5031 EnrichedLog netcoreapp3.1 117μs 70.3ns 272ns 0 0 0 4.21 KB
#5031 EnrichedLog net472 147μs 81.4ns 315ns 0.659 0.22 0 4.39 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.13μs 0.84ns 3.25ns 0.0298 0 0 2.13 KB
master EnrichedLog netcoreapp3.1 4.18μs 1.4ns 5.41ns 0.0293 0 0 2.13 KB
master EnrichedLog net472 4.79μs 2.46ns 9.53ns 0.309 0 0 1.95 KB
#5031 EnrichedLog net6.0 2.92μs 1.16ns 4.34ns 0.0293 0 0 2.13 KB
#5031 EnrichedLog netcoreapp3.1 4.13μs 1.86ns 7.19ns 0.0286 0 0 2.13 KB
#5031 EnrichedLog net472 4.92μs 2.46ns 9.19ns 0.309 0 0 1.95 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.26μs 0.708ns 2.74ns 0.0156 0 0 1.1 KB
master SendReceive netcoreapp3.1 1.84μs 1.66ns 6.42ns 0.0147 0 0 1.1 KB
master SendReceive net472 2.19μs 2.87ns 10.7ns 0.177 0 0 1.12 KB
#5031 SendReceive net6.0 1.31μs 1.54ns 5.95ns 0.0157 0 0 1.1 KB
#5031 SendReceive netcoreapp3.1 1.71μs 0.387ns 1.34ns 0.0149 0 0 1.1 KB
#5031 SendReceive net472 2.13μs 1.35ns 4.86ns 0.177 0 0 1.12 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.82μs 0.894ns 3.46ns 0.0211 0 0 1.53 KB
master EnrichedLog netcoreapp3.1 3.91μs 1.2ns 4.5ns 0.0198 0 0 1.58 KB
master EnrichedLog net472 4.31μs 0.795ns 3.08ns 0.311 0 0 1.97 KB
#5031 EnrichedLog net6.0 2.9μs 1.04ns 4.03ns 0.0215 0 0 1.53 KB
#5031 EnrichedLog netcoreapp3.1 3.92μs 2.14ns 8.28ns 0.0214 0 0 1.58 KB
#5031 EnrichedLog net472 4.21μs 2.18ns 8.43ns 0.311 0 0 1.97 KB
Benchmarks.Trace.SpanBenchmark - Slower :warning: Same allocations :heavy_check_mark:

Slower :warning: in #5031

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0 1.176 548.98 645.71

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 449ns 0.101ns 0.379ns 0.00744 0 0 536 B
master StartFinishSpan netcoreapp3.1 750ns 0.148ns 0.554ns 0.00701 0 0 536 B
master StartFinishSpan net472 707ns 0.155ns 0.579ns 0.0852 0 0 538 B
master StartFinishScope net6.0 549ns 0.247ns 0.923ns 0.00922 0 0 656 B
master StartFinishScope netcoreapp3.1 815ns 0.411ns 1.59ns 0.00869 0 0 656 B
master StartFinishScope net472 993ns 0.539ns 2.09ns 0.0978 0 0 618 B
#5031 StartFinishSpan net6.0 470ns 0.207ns 0.745ns 0.00757 0 0 536 B
#5031 StartFinishSpan netcoreapp3.1 746ns 2.8ns 10.8ns 0.00708 0 0 536 B
#5031 StartFinishSpan net472 782ns 0.782ns 3.03ns 0.0853 0 0 538 B
#5031 StartFinishScope net6.0 646ns 0.396ns 1.53ns 0.00915 0 0 656 B
#5031 StartFinishScope netcoreapp3.1 813ns 0.489ns 1.9ns 0.00884 0 0 656 B
#5031 StartFinishScope net472 978ns 0.372ns 1.39ns 0.098 0 0 618 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Slower :warning: Same allocations :heavy_check_mark:

Slower :warning: in #5031

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0 1.157 607.75 702.87

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 608ns 0.476ns 1.84ns 0.00926 0 0 656 B
master RunOnMethodBegin netcoreapp3.1 982ns 0.31ns 1.12ns 0.00863 0 0 656 B
master RunOnMethodBegin net472 1.11μs 0.391ns 1.51ns 0.098 0 0 618 B
#5031 RunOnMethodBegin net6.0 703ns 0.243ns 0.939ns 0.00914 0 0 656 B
#5031 RunOnMethodBegin netcoreapp3.1 986ns 0.498ns 1.93ns 0.00882 0 0 656 B
#5031 RunOnMethodBegin net472 1.09μs 0.212ns 0.765ns 0.0979 0 0 618 B

andrewlock avatar Jan 08 '24 19:01 andrewlock

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5031) (11.352M)   : 0, 11351789
    master (11.632M)   : 0, 11631618
    benchmarks/2.9.0 (11.757M)   : 0, 11756801

    section Automatic
    This PR (5031) (7.941M)   : 0, 7941299
    master (8.011M)   : 0, 8010820
    benchmarks/2.9.0 (8.573M)   : 0, 8573327

    section Trace stats
    This PR (5031) (8.405M)   : 0, 8405138
    master (8.335M)   : 0, 8335191

    section Manual
    This PR (5031) (9.896M)   : 0, 9896476
    master (10.240M)   : 0, 10240464

    section Manual + Automatic
    This PR (5031) (7.453M)   : 0, 7452617
    master (7.615M)   : 0, 7614917

    section Version Conflict
    This PR (5031) (6.752M)   : 0, 6751801
    master (6.860M)   : 0, 6859931


gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5031) (9.689M)   : 0, 9689028
    master (9.594M)   : 0, 9594448
    benchmarks/2.9.0 (9.605M)   : 0, 9605360

    section Automatic
    This PR (5031) (6.649M)   : 0, 6648901
    master (6.570M)   : 0, 6569521

    section Trace stats
    This PR (5031) (6.762M)   : 0, 6762365
    master (6.865M)   : 0, 6865257

    section Manual
    This PR (5031) (8.354M)   : 0, 8353704
    master (8.346M)   : 0, 8345546

    section Manual + Automatic
    This PR (5031) (6.294M)   : 0, 6293761
    master (6.111M)   : 0, 6110796

    section Version Conflict
    This PR (5031) (5.766M)   : 0, 5766328
    master (5.680M)   : 0, 5679792


gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5031) (9.982M)   : 0, 9981620
    master (10.186M)   : 0, 10186222
    benchmarks/2.9.0 (9.956M)   : 0, 9956027

    section Automatic
    This PR (5031) (7.077M)   : 0, 7077078
    master (7.005M)   : 0, 7005346
    benchmarks/2.9.0 (7.438M)   : 0, 7437608

    section Trace stats
    This PR (5031) (7.476M)   : 0, 7476250
    master (7.461M)   : 0, 7461480

    section Manual
    This PR (5031) (8.790M)   : 0, 8790144
    master (8.893M)   : 0, 8893402

    section Manual + Automatic
    This PR (5031) (6.818M)   : 0, 6817532
    master (6.733M)   : 0, 6733115

    section Version Conflict
    This PR (5031) (6.158M)   : 0, 6158028
    master (6.183M)   : 0, 6183301


gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    master (7.421M)   : 0, 7421031
    benchmarks/2.9.0 (7.813M)   : 0, 7813475

    section No attack
    master (1.758M)   : 0, 1758315
    benchmarks/2.9.0 (3.251M)   : 0, 3251456

    section Attack
    master (1.406M)   : 0, 1405570
    benchmarks/2.9.0 (2.577M)   : 0, 2577462

    section Blocking
    master (3.114M)   : 0, 3114389

    section IAST default
    master (6.585M)   : 0, 6584825

    section IAST full
    master (5.824M)   : 0, 5824226

    section Base vuln
    master (0.971M)   : 0, 971422

    section IAST vuln
    master (0.888M)   : 0, 888441

andrewlock avatar Jan 11 '24 17:01 andrewlock