dd-trace-dotnet icon indicating copy to clipboard operation
dd-trace-dotnet copied to clipboard
verified publisher icon DataDog

[Profiler] async-profiler WIP

Open gleocadie opened this issue 2 years ago • 6 comments

Summary of changes

Reason for change

Implementation details

Test coverage

Other details

gleocadie avatar Dec 12 '23 09:12 gleocadie

Datadog Report

Branch report: gleocadie/use-async-profiler-to-unwind-callstacks Commit report: 1d52055

:x: dd-trace-dotnet: 64 Failed (0 Known Flaky), 0 New Flaky, 299331 Passed, 1094 Skipped, 34m 47.64s Wall Time

:x: Failed Tests (64)

This report shows up to 5 failed tests.

  • MeasureAllocations - Datadog.Profiler.IntegrationTests.Allocations.AllocationsProfilerTest - Details

    Expand for error 
    ssert.True() Failure
xpected: True
ctual:   False

  • MeasureAllocations - Datadog.Profiler.IntegrationTests.Allocations.AllocationsProfilerTest - Details

    Expand for error 
    ssert.True() Failure
xpected: True
ctual:   False

  • ShouldGetAllocationSamples - Datadog.Profiler.IntegrationTests.Allocations.AllocationsProfilerTest - Details

    Expand for error 
    xpected allocationSamples not to be empty.

  • ShouldGetAllocationSamples - Datadog.Profiler.IntegrationTests.Allocations.AllocationsProfilerTest - Details

    Expand for error 
    xpected allocationSamples not to be empty.

  • CheckGitMetataFromBinary - Datadog.Profiler.IntegrationTests.DebugInfo.GitMetadataTest - Details

    Expand for error 
    xit code of "catchsegv" should be 0 instead of 139 (= 0x8B)
xpected: True
ctual:   False

datadog-ddstaging[bot] avatar Dec 12 '23 11:12 datadog-ddstaging[bot]

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4966) - mean (73ms)  : 63, 82
     .   : milestone, 73,
    master - mean (71ms)  : 64, 77
     .   : milestone, 71,

    section CallTarget+Inlining+NGEN
    This PR (4966) - mean (1,030ms)  : 1009, 1051
     .   : milestone, 1030,
    master - mean (1,048ms)  : 1031, 1066
     .   : milestone, 1048,


gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4966) - mean (105ms)  : 100, 110
     .   : milestone, 105,
    master - mean (106ms)  : 103, 109
     .   : milestone, 106,

    section CallTarget+Inlining+NGEN
    This PR (4966) - mean (729ms)  : 708, 750
     .   : milestone, 729,
    master - mean (741ms)  : 724, 759
     .   : milestone, 741,


gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4966) - mean (89ms)  : 86, 92
     .   : milestone, 89,
    master - mean (90ms)  : 88, 92
     .   : milestone, 90,

    section CallTarget+Inlining+NGEN
    This PR (4966) - mean (692ms)  : 666, 717
     .   : milestone, 692,
    master - mean (701ms)  : 681, 722
     .   : milestone, 701,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4966) - mean (187ms)  : 185, 190
     .   : milestone, 187,
    master - mean (187ms)  : 185, 190
     .   : milestone, 187,

    section CallTarget+Inlining+NGEN
    This PR (4966) - mean (1,145ms)  : 1131, 1159
     .   : milestone, 1145,
    master - mean (1,146ms)  : 1132, 1161
     .   : milestone, 1146,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4966) - mean (271ms)  : 266, 276
     .   : milestone, 271,
    master - mean (270ms)  : 267, 274
     .   : milestone, 270,

    section CallTarget+Inlining+NGEN
    This PR (4966) - mean (1,107ms)  : 1089, 1124
     .   : milestone, 1107,
    master - mean (1,099ms)  : 1072, 1127
     .   : milestone, 1099,


gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (4966) - mean (267ms)  : 239, 296
     .   : milestone, 267,
    master - mean (261ms)  : 256, 266
     .   : milestone, 261,

    section CallTarget+Inlining+NGEN
    This PR (4966) - mean (1,066ms)  : 1039, 1093
     .   : milestone, 1066,
    master - mean (1,067ms)  : 1040, 1094
     .   : milestone, 1067,

andrewlock avatar Dec 12 '23 11:12 andrewlock

Benchmarks Report :snail:

Benchmarks for #4966 compared to master:

  • 2 benchmarks are faster, with geometric mean 1.207
  • 4 benchmarks are slower, with geometric mean 1.141
  • 2 benchmarks have fewer allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.8μs 48.9ns 301ns 0.034 0.017 0 7.48 KB
master StartStopWithChild netcoreapp3.1 11μs 61.2ns 397ns 0.0387 0.0166 0 7.57 KB
master StartStopWithChild net472 16.8μs 78.2ns 303ns 1.35 0.356 0.124 7.95 KB
#4966 StartStopWithChild net6.0 8.46μs 34.5ns 134ns 0.0342 0.0171 0 7.48 KB
#4966 StartStopWithChild netcoreapp3.1 10.9μs 57.5ns 299ns 0.0261 0.0104 0 7.57 KB
#4966 StartStopWithChild net472 17.1μs 52.3ns 202ns 1.32 0.359 0.102 7.95 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 468μs 189ns 732ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 626μs 333ns 1.29μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 779μs 240ns 930ns 0.388 0 0 3.3 KB
#4966 WriteAndFlushEnrichedTraces net6.0 454μs 249ns 965ns 0 0 0 2.7 KB
#4966 WriteAndFlushEnrichedTraces netcoreapp3.1 629μs 248ns 927ns 0 0 0 2.7 KB
#4966 WriteAndFlushEnrichedTraces net472 783μs 187ns 700ns 0.391 0 0 3.3 KB
Benchmarks.Trace.Asm.AppSecBodyBenchmark - Faster :tada: Same allocations :heavy_check_mark:

Faster :tada: in #4966

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecBodyBenchmark.ObjectExtractorSimpleBody‑net472 1.171 193.87 165.51

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 38.5μs 28.9ns 112ns 0.0193 0 0 1.77 KB
master AllCycleSimpleBody netcoreapp3.1 41.4μs 90.8ns 340ns 0.0205 0 0 1.74 KB
master AllCycleSimpleBody net472 44.3μs 15.7ns 60.9ns 0.286 0 0 1.81 KB
master AllCycleMoreComplexBody net6.0 202μs 76.8ns 298ns 0.1 0 0 9.25 KB
master AllCycleMoreComplexBody netcoreapp3.1 214μs 111ns 431ns 0.105 0 0 9.14 KB
master AllCycleMoreComplexBody net472 226μs 107ns 413ns 1.47 0 0 9.32 KB
master ObjectExtractorSimpleBody net6.0 138ns 0.0573ns 0.214ns 0.00392 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 191ns 0.0921ns 0.344ns 0.00367 0 0 272 B
master ObjectExtractorSimpleBody net472 194ns 0.097ns 0.376ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 2.97μs 1.47ns 5.52ns 0.0521 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 3.92μs 1.07ns 4.16ns 0.0491 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 3.86μs 3.05ns 11.4ns 0.602 0.00577 0 3.8 KB
#4966 AllCycleSimpleBody net6.0 38.4μs 12.6ns 48.6ns 0.0193 0 0 1.77 KB
#4966 AllCycleSimpleBody netcoreapp3.1 41.6μs 52.5ns 203ns 0.0207 0 0 1.74 KB
#4966 AllCycleSimpleBody net472 44.6μs 13.8ns 51.8ns 0.268 0 0 1.81 KB
#4966 AllCycleMoreComplexBody net6.0 202μs 60ns 233ns 0.1 0 0 9.25 KB
#4966 AllCycleMoreComplexBody netcoreapp3.1 214μs 156ns 584ns 0.106 0 0 9.14 KB
#4966 AllCycleMoreComplexBody net472 226μs 67.3ns 252ns 1.47 0 0 9.32 KB
#4966 ObjectExtractorSimpleBody net6.0 146ns 0.0404ns 0.151ns 0.00392 0 0 280 B
#4966 ObjectExtractorSimpleBody netcoreapp3.1 195ns 0.0597ns 0.224ns 0.00374 0 0 272 B
#4966 ObjectExtractorSimpleBody net472 166ns 0.248ns 0.96ns 0.0446 0 0 281 B
#4966 ObjectExtractorMoreComplexBody net6.0 3.02μs 0.708ns 2.55ns 0.0528 0 0 3.78 KB
#4966 ObjectExtractorMoreComplexBody netcoreapp3.1 4.05μs 1.8ns 6.97ns 0.0506 0 0 3.69 KB
#4966 ObjectExtractorMoreComplexBody net472 3.84μs 1.49ns 5.39ns 0.603 0.0058 0 3.8 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Slower :warning: Same allocations :heavy_check_mark:

Slower :warning: in #4966

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Asm.AppSecWafBenchmark.RunWaf(args: NestedMap (100))‑net6.0 1.138 96,757.22 110,071.72

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWaf(args=NestedMap (10)) net6.0 51.9μs 8.21ns 28.5ns 0.233 0 0 16.06 KB
master RunWaf(args=NestedMap (10)) netcoreapp3.1 70.2μs 345ns 1.46μs 0.195 0 0 16.06 KB
master RunWaf(args=NestedMap (10)) net472 96μs 41.1ns 154ns 2.53 0.0955 0 16.14 KB
master RunWafTwice(args=NestedMap (10)) net6.0 58.2μs 333ns 2.67μs 0.214 0 0 16.6 KB
master RunWafTwice(args=NestedMap (10)) netcoreapp3.1 72.9μs 188ns 728ns 0.215 0 0 16.58 KB
master RunWafTwice(args=NestedMap (10)) net472 108μs 76.5ns 286ns 2.65 0.104 0 16.69 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 116μs 279ns 1.08μs 0.286 0 0 22.41 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 131μs 561ns 2.17μs 0.261 0 0 22.36 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 165μs 370ns 1.43μs 3.59 0.163 0 22.7 KB
master RunWaf(args=NestedMap (100)) net6.0 96.8μs 29.1ns 109ns 0.439 0 0 32.76 KB
master RunWaf(args=NestedMap (100)) netcoreapp3.1 132μs 531ns 2.06μs 0.451 0 0 33.33 KB
master RunWaf(args=NestedMap (100)) net472 187μs 110ns 412ns 5.33 0.374 0 33.67 KB
master RunWafTwice(args=NestedMap (100)) net6.0 111μs 612ns 3.67μs 0.456 0 0 33.3 KB
master RunWafTwice(args=NestedMap (100)) netcoreapp3.1 141μs 516ns 2μs 0.416 0 0 33.86 KB
master RunWafTwice(args=NestedMap (100)) net472 194μs 105ns 393ns 5.35 0.389 0 34.23 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 164μs 54.7ns 205ns 0.553 0 0 39.1 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 206μs 899ns 3.48μs 0.49 0 0 39.63 KB
master RunWafWithAttack(args=Neste(...)tack) [23]) net472 258μs 136ns 527ns 6.37 0.52 0 40.23 KB
master RunWaf(args=NestedMap (20)) net6.0 97.8μs 33.8ns 131ns 0.428 0 0 32.18 KB
master RunWaf(args=NestedMap (20)) netcoreapp3.1 137μs 556ns 2.08μs 0.389 0 0 32.3 KB
master RunWaf(args=NestedMap (20)) net472 185μs 183ns 684ns 5.17 0.369 0 32.63 KB
master RunWafTwice(args=NestedMap (20)) net6.0 101μs 48.7ns 188ns 0.448 0 0 32.72 KB
master RunWafTwice(args=NestedMap (20)) netcoreapp3.1 147μs 222ns 861ns 0.419 0 0 32.82 KB
master RunWafTwice(args=NestedMap (20)) net472 196μs 781ns 3.03μs 5.25 0.389 0 33.19 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 166μs 593ns 2.3μs 0.476 0 0 38.53 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 201μs 1.06μs 5.41μs 0.497 0 0 38.6 KB
master RunWafWithAttack(args=Neste(...)tack) [22]) net472 263μs 137ns 531ns 6.12 0.51 0 39.2 KB
#4966 RunWaf(args=NestedMap (10)) net6.0 50.6μs 14.8ns 57.3ns 0.221 0 0 16.06 KB
#4966 RunWaf(args=NestedMap (10)) netcoreapp3.1 69.9μs 168ns 652ns 0.204 0 0 16.06 KB
#4966 RunWaf(args=NestedMap (10)) net472 101μs 64.9ns 251ns 2.52 0.0951 0 16.14 KB
#4966 RunWafTwice(args=NestedMap (10)) net6.0 56μs 290ns 1.48μs 0.211 0 0 16.6 KB
#4966 RunWafTwice(args=NestedMap (10)) netcoreapp3.1 72.4μs 332ns 1.28μs 0.207 0 0 16.58 KB
#4966 RunWafTwice(args=NestedMap (10)) net472 103μs 74.6ns 289ns 2.62 0.103 0 16.69 KB
#4966 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 107μs 35ns 153ns 0.322 0 0 22.41 KB
#4966 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 130μs 650ns 3.05μs 0.264 0 0 22.36 KB
#4966 RunWafWithAttack(args=Neste(...)tack) [22]) net472 158μs 52.2ns 202ns 3.55 0.158 0 22.7 KB
#4966 RunWaf(args=NestedMap (100)) net6.0 110μs 133ns 514ns 0.433 0 0 32.76 KB
#4966 RunWaf(args=NestedMap (100)) netcoreapp3.1 134μs 704ns 3.59μs 0.414 0 0 33.33 KB
#4966 RunWaf(args=NestedMap (100)) net472 192μs 707ns 2.74μs 5.3 0.379 0 33.67 KB
#4966 RunWafTwice(args=NestedMap (100)) net6.0 110μs 635ns 5.31μs 0.457 0 0 33.3 KB
#4966 RunWafTwice(args=NestedMap (100)) netcoreapp3.1 138μs 700ns 3.05μs 0.408 0 0 33.86 KB
#4966 RunWafTwice(args=NestedMap (100)) net472 198μs 936ns 3.62μs 5.35 0.389 0 34.23 KB
#4966 RunWafWithAttack(args=Neste(...)tack) [23]) net6.0 158μs 45.5ns 176ns 0.55 0 0 39.1 KB
#4966 RunWafWithAttack(args=Neste(...)tack) [23]) netcoreapp3.1 199μs 568ns 2.2μs 0.494 0 0 39.63 KB
#4966 RunWafWithAttack(args=Neste(...)tack) [23]) net472 260μs 704ns 2.73μs 6.29 0.513 0 40.23 KB
#4966 RunWaf(args=NestedMap (20)) net6.0 108μs 421ns 1.63μs 0.445 0 0 32.18 KB
#4966 RunWaf(args=NestedMap (20)) netcoreapp3.1 132μs 734ns 4.64μs 0.401 0 0 32.3 KB
#4966 RunWaf(args=NestedMap (20)) net472 188μs 147ns 571ns 5.17 0.376 0 32.63 KB
#4966 RunWafTwice(args=NestedMap (20)) net6.0 105μs 522ns 2.15μs 0.431 0 0 32.72 KB
#4966 RunWafTwice(args=NestedMap (20)) netcoreapp3.1 139μs 679ns 2.8μs 0.427 0 0 32.82 KB
#4966 RunWafTwice(args=NestedMap (20)) net472 193μs 95.6ns 370ns 5.21 0.386 0 33.19 KB
#4966 RunWafWithAttack(args=Neste(...)tack) [22]) net6.0 157μs 70.7ns 274ns 0.552 0 0 38.53 KB
#4966 RunWafWithAttack(args=Neste(...)tack) [22]) netcoreapp3.1 203μs 1.1μs 6.13μs 0.507 0 0 38.6 KB
#4966 RunWafWithAttack(args=Neste(...)tack) [22]) net472 262μs 44.8ns 162ns 6.11 0.509 0 39.2 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 172μs 104ns 404ns 0.172 0 0 18.25 KB
master SendRequest netcoreapp3.1 192μs 220ns 794ns 0.189 0 0 20.41 KB
master SendRequest net472 0.00105ns 0.000296ns 0.00115ns 0 0 0 0 b
#4966 SendRequest net6.0 168μs 86ns 322ns 0.168 0 0 18.25 KB
#4966 SendRequest netcoreapp3.1 194μs 161ns 603ns 0.193 0 0 20.42 KB
#4966 SendRequest net472 0.000633ns 0.000304ns 0.00114ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: Fewer allocations :tada:

Fewer allocations :tada: in #4966

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑net6.0 41.77 KB 41.46 KB -309 B -0.74%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 558μs 1.22μs 4.73μs 0.275 0 0 41.77 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 653μs 851ns 3.29μs 0.324 0 0 41.77 KB
master WriteAndFlushEnrichedTraces net472 843μs 2.6μs 9.72μs 8.45 2.53 0.422 53.25 KB
#4966 WriteAndFlushEnrichedTraces net6.0 562μs 1.48μs 5.73μs 0.551 0 0 41.46 KB
#4966 WriteAndFlushEnrichedTraces netcoreapp3.1 638μs 1.1μs 4.26μs 0.321 0 0 41.76 KB
#4966 WriteAndFlushEnrichedTraces net472 841μs 3.95μs 15.8μs 8.3 2.62 0.437 53.21 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.15μs 0.787ns 3.05ns 0.0104 0 0 768 B
master ExecuteNonQuery netcoreapp3.1 1.52μs 0.663ns 2.57ns 0.00991 0 0 768 B
master ExecuteNonQuery net472 1.81μs 1.29ns 4.98ns 0.116 0 0 730 B
#4966 ExecuteNonQuery net6.0 1.04μs 0.554ns 2.14ns 0.0109 0 0 768 B
#4966 ExecuteNonQuery netcoreapp3.1 1.48μs 0.825ns 3.19ns 0.0102 0 0 768 B
#4966 ExecuteNonQuery net472 1.8μs 0.647ns 2.51ns 0.115 0 0 730 B
Benchmarks.Trace.ElasticsearchBenchmark - Slower :warning: Same allocations :heavy_check_mark:

Slower :warning: in #4966

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.ElasticsearchBenchmark.CallElasticsearch‑net6.0 1.136 1,167.33 1,325.89

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.17μs 0.395ns 1.48ns 0.013 0 0 936 B
master CallElasticsearch netcoreapp3.1 1.54μs 0.775ns 2.9ns 0.0123 0 0 936 B
master CallElasticsearch net472 2.62μs 0.369ns 1.33ns 0.151 0 0 955 B
master CallElasticsearchAsync net6.0 1.25μs 0.354ns 1.32ns 0.0126 0 0 912 B
master CallElasticsearchAsync netcoreapp3.1 1.62μs 1.58ns 5.92ns 0.0129 0 0 984 B
master CallElasticsearchAsync net472 2.77μs 0.931ns 3.6ns 0.16 0 0 1.01 KB
#4966 CallElasticsearch net6.0 1.33μs 0.575ns 2.15ns 0.0133 0 0 936 B
#4966 CallElasticsearch netcoreapp3.1 1.57μs 0.771ns 2.99ns 0.0126 0 0 936 B
#4966 CallElasticsearch net472 2.62μs 2.06ns 7.96ns 0.151 0.0013 0 955 B
#4966 CallElasticsearchAsync net6.0 1.23μs 0.437ns 1.57ns 0.0123 0 0 912 B
#4966 CallElasticsearchAsync netcoreapp3.1 1.65μs 0.634ns 2.37ns 0.0134 0 0 984 B
#4966 CallElasticsearchAsync net472 2.7μs 1.45ns 5.63ns 0.16 0 0 1.01 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.32μs 0.802ns 3.11ns 0.0125 0 0 912 B
master ExecuteAsync netcoreapp3.1 1.57μs 2.66ns 10.3ns 0.0123 0 0 912 B
master ExecuteAsync net472 1.92μs 1.03ns 3.97ns 0.138 0 0 875 B
#4966 ExecuteAsync net6.0 1.26μs 0.799ns 2.99ns 0.0126 0 0 912 B
#4966 ExecuteAsync netcoreapp3.1 1.68μs 1.23ns 4.76ns 0.0125 0 0 912 B
#4966 ExecuteAsync net472 1.75μs 0.976ns 3.78ns 0.138 0 0 875 B
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 3.98μs 1.16ns 4.47ns 0.0299 0 0 2.1 KB
master SendAsync netcoreapp3.1 4.99μs 2.54ns 8.79ns 0.0348 0 0 2.63 KB
master SendAsync net472 7.75μs 3.85ns 14.9ns 0.522 0 0 3.31 KB
#4966 SendAsync net6.0 3.93μs 1ns 3.74ns 0.0297 0 0 2.1 KB
#4966 SendAsync netcoreapp3.1 5.03μs 2.19ns 8.47ns 0.0352 0 0 2.63 KB
#4966 SendAsync net472 7.85μs 4.2ns 16.3ns 0.523 0 0 3.31 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Faster :tada: Fewer allocations :tada:

Faster :tada: in #4966

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark(parameters: System.Collections.Generic.List`1[System.String])‑net472 1.243 62,200.00 50,050.00

Fewer allocations :tada: in #4966

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark(parameters: System.Collections.Generic.List`1[System.String])‑net472 62.57 KB 54.62 KB -7.94 KB -12.70%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark(parameters=Syste(...)ring] [48]) net6.0 60.4μs 827ns 8.27μs 0 0 0 43.44 KB
master StringConcatBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 59.4μs 732ns 7.21μs 0 0 0 42.64 KB
master StringConcatBenchmark(parameters=Syste(...)ring] [48]) net472 38.2μs 77.5ns 290ns 0 0 0 62.57 KB
master StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net6.0 97.1μs 2.61μs 26.1μs 0 0 0 43.29 KB
master StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 106μs 2.26μs 22.3μs 0 0 0 42.64 KB
master StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net472 62.2μs 232ns 838ns 0 0 0 57.34 KB
#4966 StringConcatBenchmark(parameters=Syste(...)ring] [48]) net6.0 62.2μs 893ns 8.88μs 0 0 0 43.44 KB
#4966 StringConcatBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 60.8μs 759ns 7.44μs 0 0 0 42.64 KB
#4966 StringConcatBenchmark(parameters=Syste(...)ring] [48]) net472 36.9μs 103ns 384ns 0 0 0 54.62 KB
#4966 StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net6.0 94.8μs 2.58μs 25.8μs 0 0 0 43.29 KB
#4966 StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) netcoreapp3.1 107μs 3.54μs 35.2μs 0 0 0 42.64 KB
#4966 StringConcatAspectBenchmark(parameters=Syste(...)ring] [48]) net472 50.5μs 279ns 1.65μs 0 0 0 57.34 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.49μs 0.714ns 2.67ns 0.0221 0 0 1.57 KB
master EnrichedLog netcoreapp3.1 2.33μs 1.65ns 5.94ns 0.0209 0 0 1.57 KB
master EnrichedLog net472 2.48μs 1.63ns 6.31ns 0.238 0 0 1.5 KB
#4966 EnrichedLog net6.0 1.52μs 0.949ns 3.42ns 0.022 0 0 1.57 KB
#4966 EnrichedLog netcoreapp3.1 2.22μs 0.894ns 3.35ns 0.0211 0 0 1.57 KB
#4966 EnrichedLog net472 2.57μs 3.57ns 13.4ns 0.238 0 0 1.5 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 113μs 151ns 585ns 0.0561 0 0 4.21 KB
master EnrichedLog netcoreapp3.1 118μs 113ns 438ns 0.0596 0 0 4.21 KB
master EnrichedLog net472 151μs 250ns 969ns 0.672 0.224 0 4.39 KB
#4966 EnrichedLog net6.0 113μs 113ns 407ns 0.0564 0 0 4.21 KB
#4966 EnrichedLog netcoreapp3.1 118μs 175ns 678ns 0 0 0 4.21 KB
#4966 EnrichedLog net472 147μs 90.7ns 351ns 0.661 0.22 0 4.39 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.89μs 1.06ns 3.95ns 0.0292 0 0 2.13 KB
master EnrichedLog netcoreapp3.1 4.1μs 2.75ns 10.3ns 0.0275 0 0 2.13 KB
master EnrichedLog net472 4.86μs 1.51ns 5.85ns 0.308 0 0 1.95 KB
#4966 EnrichedLog net6.0 3.1μs 0.926ns 3.59ns 0.0296 0 0 2.13 KB
#4966 EnrichedLog netcoreapp3.1 4.3μs 2.63ns 10.2ns 0.0279 0 0 2.13 KB
#4966 EnrichedLog net472 4.9μs 2.17ns 8.42ns 0.307 0 0 1.95 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.4μs 0.674ns 2.52ns 0.0154 0 0 1.1 KB
master SendReceive netcoreapp3.1 1.79μs 0.501ns 1.94ns 0.0143 0 0 1.1 KB
master SendReceive net472 2.11μs 2.33ns 8.39ns 0.177 0 0 1.12 KB
#4966 SendReceive net6.0 1.5μs 1.91ns 7.4ns 0.0156 0 0 1.1 KB
#4966 SendReceive netcoreapp3.1 1.73μs 0.738ns 2.76ns 0.0147 0 0 1.1 KB
#4966 SendReceive net472 2.14μs 3.44ns 13.3ns 0.176 0 0 1.12 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.85μs 0.649ns 2.43ns 0.0214 0 0 1.53 KB
master EnrichedLog netcoreapp3.1 3.92μs 1.64ns 6.37ns 0.0214 0 0 1.58 KB
master EnrichedLog net472 4.38μs 1.2ns 4.64ns 0.312 0 0 1.97 KB
#4966 EnrichedLog net6.0 2.87μs 0.961ns 3.6ns 0.0202 0 0 1.53 KB
#4966 EnrichedLog netcoreapp3.1 3.85μs 1.66ns 6.21ns 0.0211 0 0 1.58 KB
#4966 EnrichedLog net472 4.23μs 1.26ns 4.9ns 0.312 0 0 1.97 KB
Benchmarks.Trace.SpanBenchmark - Slower :warning: Same allocations :heavy_check_mark:

Slower :warning: in #4966

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 1.180 460.84 543.67
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net472 1.114 938.25 1,044.82

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 461ns 0.286ns 1.07ns 0.00741 0 0 536 B
master StartFinishSpan netcoreapp3.1 683ns 0.592ns 2.29ns 0.00734 0 0 536 B
master StartFinishSpan net472 747ns 0.64ns 2.48ns 0.0853 0 0 538 B
master StartFinishScope net6.0 652ns 1.49ns 5.78ns 0.00912 0 0 656 B
master StartFinishScope netcoreapp3.1 907ns 3.31ns 12.8ns 0.00886 0 0 656 B
master StartFinishScope net472 937ns 0.71ns 2.75ns 0.098 0 0 618 B
#4966 StartFinishSpan net6.0 543ns 0.447ns 1.73ns 0.00749 0 0 536 B
#4966 StartFinishSpan netcoreapp3.1 690ns 0.226ns 0.846ns 0.00724 0 0 536 B
#4966 StartFinishSpan net472 750ns 0.382ns 1.48ns 0.0852 0 0 538 B
#4966 StartFinishScope net6.0 645ns 0.693ns 2.68ns 0.00922 0 0 656 B
#4966 StartFinishScope netcoreapp3.1 830ns 0.342ns 1.32ns 0.00893 0 0 656 B
#4966 StartFinishScope net472 1.05μs 0.439ns 1.7ns 0.098 0 0 618 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 610ns 0.176ns 0.633ns 0.00919 0 0 656 B
master RunOnMethodBegin netcoreapp3.1 939ns 1.5ns 5.41ns 0.00876 0 0 656 B
master RunOnMethodBegin net472 1.06μs 0.391ns 1.51ns 0.0977 0 0 618 B
#4966 RunOnMethodBegin net6.0 583ns 0.15ns 0.562ns 0.0093 0 0 656 B
#4966 RunOnMethodBegin netcoreapp3.1 955ns 0.491ns 1.9ns 0.00894 0 0 656 B
#4966 RunOnMethodBegin net472 1.13μs 0.414ns 1.49ns 0.0981 0 0 618 B

andrewlock avatar Dec 12 '23 13:12 andrewlock

Datadog Report

Branch report: gleocadie/use-async-profiler-to-unwind-callstacks Commit report: d0fb6ba

:x: dd-trace-dotnet: 10 Failed (0 Known Flaky), 0 New Flaky, 296944 Passed, 1034 Skipped, 48m 0.69s Wall Time

:x: Failed Tests (10)

This report shows up to 5 failed tests.

  • CheckTimestampAsLabel - Datadog.Profiler.IntegrationTests.Timeline.TimelineTest

  • CheckTimestampAsLabel - Datadog.Profiler.IntegrationTests.Timeline.TimelineTest

  • CheckTimestampAsLabel - Datadog.Profiler.IntegrationTests.Timeline.TimelineTest - Details

    Expand for error 
    he test Samples.BuggyBits is running for too long or was lost

  • CheckTimestampAsLabel - Datadog.Profiler.IntegrationTests.Timeline.TimelineTest - Details

    Expand for error 
    he test Samples.BuggyBits is running for too long or was lost

  • CheckTimestampAsLabel - Datadog.Profiler.IntegrationTests.Timeline.TimelineTest - Details

    Expand for error 
    he test Samples.BuggyBits is running for too long or was lost

datadog-ddstaging[bot] avatar Dec 19 '23 12:12 datadog-ddstaging[bot]

Datadog Report

Branch report: gleocadie/use-async-profiler-to-unwind-callstacks Commit report: 368f4da Test service: dd-trace-dotnet

:x: 7 Failed (0 Known Flaky), 272501 Passed, 1213 Skipped, 30m 30.37s Wall Time

:x: Failed Tests (7)

This report shows up to 5 failed tests.

  • CheckExporterDoesUsePipeNameEvenIfItDoesNotExist - Datadog.Profiler.IntegrationTests.WindowsOnly.NamedPipeTestcs - Details

    Expand for error 
    equence contains more than one matching element

  • CheckExporterDoesUsePipeNameEvenIfItDoesNotExist - Datadog.Profiler.IntegrationTests.WindowsOnly.NamedPipeTestcs - Details

    Expand for error 
    equence contains more than one matching element

  • CheckExporterDoesUsePipeNameEvenIfItDoesNotExist - Datadog.Profiler.IntegrationTests.WindowsOnly.NamedPipeTestcs - Details

    Expand for error 
    equence contains more than one matching element

  • CheckExporterDoesUsePipeNameEvenIfItDoesNotExist - Datadog.Profiler.IntegrationTests.WindowsOnly.NamedPipeTestcs - Details

    Expand for error 
    equence contains more than one matching element

  • CheckExporterDoesUsePipeNameEvenIfItDoesNotExist - Datadog.Profiler.IntegrationTests.WindowsOnly.NamedPipeTestcs - Details

    Expand for error 
    equence contains more than one matching element

datadog-ddstaging[bot] avatar Jan 08 '24 22:01 datadog-ddstaging[bot]

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4966) (11.784M)   : 0, 11784318
    master (11.338M)   : 0, 11337986
    benchmarks/2.9.0 (11.276M)   : 0, 11275582

    section Automatic
    This PR (4966) (8.092M)   : 0, 8092205
    master (7.890M)   : 0, 7889655
    benchmarks/2.9.0 (8.217M)   : 0, 8216570

    section Trace stats
    This PR (4966) (8.379M)   : 0, 8378609
    master (8.250M)   : 0, 8249501

    section Manual
    This PR (4966) (10.328M)   : 0, 10328164
    master (10.066M)   : 0, 10065780

    section Manual + Automatic
    This PR (4966) (7.673M)   : 0, 7673418
    master (7.457M)   : 0, 7457332

    section Version Conflict
    This PR (4966) (6.921M)   : 0, 6921279
    master (6.713M)   : 0, 6712794


gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4966) (9.636M)   : 0, 9636257
    master (9.532M)   : 0, 9532434
    benchmarks/2.9.0 (9.573M)   : 0, 9572514

    section Automatic
    This PR (4966) (6.585M)   : 0, 6585304
    master (6.527M)   : 0, 6527082

    section Trace stats
    This PR (4966) (6.938M)   : 0, 6937506
    master (6.925M)   : 0, 6925177

    section Manual
    This PR (4966) (8.351M)   : 0, 8350929
    master (8.425M)   : 0, 8424666

    section Manual + Automatic
    This PR (4966) (6.127M)   : 0, 6126504
    master (6.241M)   : 0, 6240781

    section Version Conflict
    This PR (4966) (5.693M)   : 0, 5692725
    master (5.675M)   : 0, 5674933


gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (4966) (10.428M)   : 0, 10427891
    master (10.353M)   : 0, 10352797
    benchmarks/2.9.0 (9.528M)   : 0, 9527529

    section Automatic
    This PR (4966) (7.119M)   : 0, 7118542
    master (7.322M)   : 0, 7322309
    benchmarks/2.9.0 (7.277M)   : 0, 7277162

    section Trace stats
    This PR (4966) (7.516M)   : 0, 7515825
    master (7.541M)   : 0, 7541304

    section Manual
    This PR (4966) (9.007M)   : 0, 9007358
    master (8.961M)   : 0, 8961262

    section Manual + Automatic
    This PR (4966) (7.072M)   : 0, 7072305
    master (7.072M)   : 0, 7071655

    section Version Conflict
    This PR (4966) (6.396M)   : 0, 6395820
    master (6.496M)   : 0, 6496389


gantt
    title Throughput Linux x64 (ASM) (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    master (7.451M)   : 0, 7451233
    benchmarks/2.9.0 (7.942M)   : 0, 7942095

    section No attack
    master (1.765M)   : 0, 1764994
    benchmarks/2.9.0 (3.272M)   : 0, 3271840

    section Attack
    master (1.412M)   : 0, 1412054
    benchmarks/2.9.0 (2.547M)   : 0, 2546976

    section Blocking
    master (3.131M)   : 0, 3130640

    section IAST default
    master (6.397M)   : 0, 6397265

    section IAST full
    master (5.712M)   : 0, 5712493

    section Base vuln
    master (0.935M)   : 0, 934917

    section IAST vuln
    master (0.910M)   : 0, 909956

andrewlock avatar Jan 10 '24 01:01 andrewlock