dd-trace-dotnet icon indicating copy to clipboard operation
dd-trace-dotnet copied to clipboard
verified publisher icon DataDog

[v3] Package musl assets in linux glibc installers

Open andrewlock opened this issue 1 year ago • 5 comments

Summary of changes

  • Adds the musl assets to the Linux glibc installers
  • Add smoke tests for using the glibc installer on alpine

Reason for change

To simplify onboarding, we now include the musl assets in the glibc installer packages. Instead of having to choose which package to install, there's a single "correct" choice.

For now, we have left the -musl.tar.gz package with only the -musl artifacts (i.e. we didn't package the glibc assets there). The reasoning was that customers have already explicitly selected musl at this point, so the glibc assets are very unlikely to be used. Alpine customers are also likely more size-conscious. We could potentially consider deprecating the artifact in a future major release, but don't plan to currently.

Implementation details

Package the musl assets in the glibc installers (in the linux-musl-x64 path). Note that the universal binaries (ld_preload, native loader) are identical, so we hardlink those in to ensure we don't increase package size too much.

Test coverage

Added smoke tests to confirm that we can install and run with the universal (glibc) package when running on alpine.

Checking the size of the assets:

  • Before: ~20MB compressed (.tar.gz etc size), ~60MB uncompressed (on disk)
  • After: ~30MB compressed (.tar.gz etc size), ~80MB uncompressed (on disk)

Other details

Currently the dd-dotnet.sh script directs customers to install the -musl package when running on Alpine (and it detects you've installed the wrong package). Should we point to the universal package instead? Bit of a moot point, as we shouldn't ever hit that branch now (if all is working well)

Requires:

  • https://github.com/DataDog/dd-trace-dotnet/pull/5580
  • https://github.com/DataDog/dd-trace-dotnet/pull/5593

andrewlock avatar May 22 '24 11:05 andrewlock

Datadog Report

Branch report: andrew/ci/glibc-musl-packaging Commit report: 1a9ae85 Test service: dd-trace-dotnet

:white_check_mark: 0 Failed, 420695 Passed, 3337 Skipped, 34h 53m 57.18s Total Time

datadog-ddstaging[bot] avatar May 22 '24 11:05 datadog-ddstaging[bot]

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5595) - mean (74ms)  : 67, 81
     .   : milestone, 74,
    master - mean (75ms)  : 63, 87
     .   : milestone, 75,

    section CallTarget+Inlining+NGEN
    This PR (5595) - mean (1,037ms)  : 1016, 1058
     .   : milestone, 1037,
    master - mean (1,021ms)  : 993, 1049
     .   : milestone, 1021,


gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5595) - mean (110ms)  : 106, 114
     .   : milestone, 110,
    master - mean (109ms)  : 108, 111
     .   : milestone, 109,

    section CallTarget+Inlining+NGEN
    This PR (5595) - mean (732ms)  : 711, 752
     .   : milestone, 732,
    master - mean (702ms)  : 680, 723
     .   : milestone, 702,


gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5595) - mean (92ms)  : 89, 96
     .   : milestone, 92,
    master - mean (93ms)  : 89, 97
     .   : milestone, 93,

    section CallTarget+Inlining+NGEN
    This PR (5595) - mean (681ms)  : 651, 711
     .   : milestone, 681,
    master - mean (663ms)  : 637, 690
     .   : milestone, 663,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5595) - mean (191ms)  : 187, 194
     .   : milestone, 191,
    master - mean (191ms)  : 187, 194
     .   : milestone, 191,

    section CallTarget+Inlining+NGEN
    This PR (5595) - mean (1,130ms)  : 1110, 1150
     .   : milestone, 1130,
    master - mean (1,109ms)  : 1087, 1131
     .   : milestone, 1109,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5595) - mean (275ms)  : 271, 280
     .   : milestone, 275,
    master - mean (276ms)  : 271, 281
     .   : milestone, 276,

    section CallTarget+Inlining+NGEN
    This PR (5595) - mean (899ms)  : 879, 919
     .   : milestone, 899,
    master - mean (881ms)  : 854, 907
     .   : milestone, 881,


gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5595) - mean (265ms)  : 261, 269
     .   : milestone, 265,
    master - mean (265ms)  : 261, 268
     .   : milestone, 265,

    section CallTarget+Inlining+NGEN
    This PR (5595) - mean (879ms)  : 848, 910
     .   : milestone, 879,
    master - mean (861ms)  : 837, 885
     .   : milestone, 861,

andrewlock avatar May 22 '24 11:05 andrewlock

Throughput/Crank Report :zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5595) (11.442M)   : 0, 11442447
    master (11.535M)   : 0, 11534927
    benchmarks/2.9.0 (11.421M)   : 0, 11421113

    section Automatic
    This PR (5595) (7.713M)   : 0, 7712986
    master (6.986M)   : 0, 6986192
    benchmarks/2.9.0 (8.159M)   : 0, 8158990

    section Trace stats
    master (8.081M)   : 0, 8081462

    section Manual
    master (9.796M)   : 0, 9795579

    section Manual + Automatic
    This PR (5595) (7.222M)   : 0, 7222375
    master (7.348M)   : 0, 7347745


gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5595) (9.454M)   : 0, 9453686
    master (9.681M)   : 0, 9681417
    benchmarks/2.9.0 (9.482M)   : 0, 9482175

    section Automatic
    This PR (5595) (6.565M)   : 0, 6564768
    master (6.035M)   : 0, 6034740

    section Trace stats
    master (6.907M)   : 0, 6907154

    section Manual
    master (8.214M)   : 0, 8213952

    section Manual + Automatic
    This PR (5595) (5.966M)   : 0, 5965886
    master (6.164M)   : 0, 6163622


gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5595) (9.805M)   : 0, 9804842
    master (9.890M)   : 0, 9889791
    benchmarks/2.9.0 (9.965M)   : 0, 9965104

    section Automatic
    This PR (5595) (6.987M)   : 0, 6986871
    master (7.142M)   : 0, 7142473
    benchmarks/2.9.0 (7.299M)   : 0, 7298728

    section Trace stats
    master (7.381M)   : 0, 7380548

    section Manual
    master (8.866M)   : 0, 8865782

    section Manual + Automatic
    This PR (5595) (6.694M)   : 0, 6694332
    master (6.917M)   : 0, 6916590

andrewlock avatar May 22 '24 11:05 andrewlock

Benchmarks Report for tracer :snail:

Benchmarks for #5595 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.135
  • 2 benchmarks are slower, with geometric mean 1.188
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 7.79μs 43.4ns 271ns 0.0149 0.00745 0 5.43 KB
master StartStopWithChild netcoreapp3.1 9.72μs 55ns 401ns 0.0186 0.0093 0 5.62 KB
master StartStopWithChild net472 15.9μs 35.4ns 132ns 1.01 0.291 0.0945 6.07 KB
#5595 StartStopWithChild net6.0 7.65μs 43.2ns 287ns 0.0122 0.00407 0 5.42 KB
#5595 StartStopWithChild netcoreapp3.1 9.65μs 53.6ns 317ns 0.0227 0.0091 0 5.62 KB
#5595 StartStopWithChild net472 16.1μs 60.7ns 235ns 1.04 0.326 0.0978 6.06 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 457μs 134ns 519ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 634μs 154ns 578ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 841μs 255ns 989ns 0.419 0 0 3.3 KB
#5595 WriteAndFlushEnrichedTraces net6.0 476μs 167ns 646ns 0 0 0 2.7 KB
#5595 WriteAndFlushEnrichedTraces netcoreapp3.1 629μs 292ns 1.05μs 0 0 0 2.7 KB
#5595 WriteAndFlushEnrichedTraces net472 837μs 278ns 1.08μs 0.417 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 173μs 248ns 959ns 0.173 0 0 18.44 KB
master SendRequest netcoreapp3.1 194μs 318ns 1.19μs 0.193 0 0 20.6 KB
master SendRequest net472 0.000295ns 0.000166ns 0.000621ns 0 0 0 0 b
#5595 SendRequest net6.0 172μs 231ns 893ns 0.175 0 0 18.44 KB
#5595 SendRequest netcoreapp3.1 194μs 326ns 1.26μs 0.192 0 0 20.6 KB
#5595 SendRequest net472 5.87E‑05ns 5.87E‑05ns 0.000212ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 555μs 1.42μs 5.5μs 0.556 0 0 41.72 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 632μs 1.13μs 3.91μs 0.316 0 0 41.75 KB
master WriteAndFlushEnrichedTraces net472 851μs 3.86μs 14.9μs 8.39 2.52 0.419 53.24 KB
#5595 WriteAndFlushEnrichedTraces net6.0 548μs 327ns 1.22μs 0.546 0 0 41.61 KB
#5595 WriteAndFlushEnrichedTraces netcoreapp3.1 649μs 1.25μs 4.85μs 0.331 0 0 41.65 KB
#5595 WriteAndFlushEnrichedTraces net472 842μs 3.34μs 13μs 8.08 2.55 0.425 53.22 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.09μs 0.404ns 1.57ns 0.0114 0 0 808 B
master ExecuteNonQuery netcoreapp3.1 1.47μs 2.22ns 8.61ns 0.0109 0 0 808 B
master ExecuteNonQuery net472 1.69μs 0.698ns 2.61ns 0.122 0 0 770 B
#5595 ExecuteNonQuery net6.0 1.12μs 0.672ns 2.6ns 0.0113 0 0 808 B
#5595 ExecuteNonQuery netcoreapp3.1 1.47μs 0.697ns 2.61ns 0.011 0 0 808 B
#5595 ExecuteNonQuery net472 1.82μs 2.25ns 8.73ns 0.122 0 0 770 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.24μs 0.609ns 2.28ns 0.0133 0 0 976 B
master CallElasticsearch netcoreapp3.1 1.61μs 0.689ns 2.58ns 0.0129 0 0 976 B
master CallElasticsearch net472 2.51μs 1.72ns 6.43ns 0.158 0 0 995 B
master CallElasticsearchAsync net6.0 1.37μs 0.657ns 2.46ns 0.0137 0 0 952 B
master CallElasticsearchAsync netcoreapp3.1 1.64μs 0.657ns 2.46ns 0.0139 0 0 1.02 KB
master CallElasticsearchAsync net472 2.65μs 2.13ns 8.26ns 0.167 0 0 1.05 KB
#5595 CallElasticsearch net6.0 1.15μs 3.84ns 13.8ns 0.0136 0 0 976 B
#5595 CallElasticsearch netcoreapp3.1 1.48μs 0.621ns 2.41ns 0.0133 0 0 976 B
#5595 CallElasticsearch net472 2.47μs 1.97ns 7.61ns 0.157 0.00123 0 995 B
#5595 CallElasticsearchAsync net6.0 1.25μs 0.587ns 2.27ns 0.0131 0 0 952 B
#5595 CallElasticsearchAsync netcoreapp3.1 1.66μs 0.896ns 3.35ns 0.0142 0 0 1.02 KB
#5595 CallElasticsearchAsync net472 2.57μs 1.6ns 6.18ns 0.166 0.00128 0 1.05 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.28μs 0.931ns 3.61ns 0.0134 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.72μs 0.925ns 3.46ns 0.0129 0 0 952 B
master ExecuteAsync net472 1.75μs 1.42ns 5.5ns 0.145 0 0 915 B
#5595 ExecuteAsync net6.0 1.39μs 0.615ns 2.38ns 0.0132 0 0 952 B
#5595 ExecuteAsync netcoreapp3.1 1.68μs 0.653ns 2.44ns 0.0126 0 0 952 B
#5595 ExecuteAsync net472 1.76μs 1ns 3.89ns 0.145 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.14μs 1.65ns 5.95ns 0.0311 0 0 2.22 KB
master SendAsync netcoreapp3.1 5.09μs 2.42ns 9.36ns 0.0383 0 0 2.76 KB
master SendAsync net472 7.65μs 2.72ns 10.5ns 0.497 0 0 3.15 KB
#5595 SendAsync net6.0 4.14μs 2ns 7.48ns 0.0311 0 0 2.22 KB
#5595 SendAsync netcoreapp3.1 5.14μs 2.44ns 9.46ns 0.036 0 0 2.76 KB
#5595 SendAsync net472 7.55μs 3.66ns 14.2ns 0.498 0 0 3.15 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.49μs 0.387ns 1.45ns 0.0232 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 2.24μs 3.75ns 14.5ns 0.0225 0 0 1.64 KB
master EnrichedLog net472 2.52μs 2.6ns 10.1ns 0.249 0 0 1.57 KB
#5595 EnrichedLog net6.0 1.45μs 0.684ns 2.56ns 0.0234 0 0 1.64 KB
#5595 EnrichedLog netcoreapp3.1 2.32μs 1.64ns 6.12ns 0.022 0 0 1.64 KB
#5595 EnrichedLog net472 2.49μs 1.32ns 5.1ns 0.249 0 0 1.57 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 119μs 136ns 526ns 0 0 0 4.28 KB
master EnrichedLog netcoreapp3.1 122μs 200ns 774ns 0 0 0 4.28 KB
master EnrichedLog net472 151μs 142ns 530ns 0.681 0.227 0 4.46 KB
#5595 EnrichedLog net6.0 117μs 118ns 459ns 0.058 0 0 4.28 KB
#5595 EnrichedLog netcoreapp3.1 121μs 222ns 858ns 0.0602 0 0 4.28 KB
#5595 EnrichedLog net472 149μs 169ns 655ns 0.667 0.222 0 4.46 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.99μs 0.97ns 3.76ns 0.0314 0 0 2.2 KB
master EnrichedLog netcoreapp3.1 4.05μs 2.29ns 8.56ns 0.0303 0 0 2.2 KB
master EnrichedLog net472 4.84μs 2.26ns 8.76ns 0.319 0 0 2.02 KB
#5595 EnrichedLog net6.0 3.02μs 0.852ns 3.19ns 0.0302 0 0 2.2 KB
#5595 EnrichedLog netcoreapp3.1 4.13μs 2.11ns 8.19ns 0.0291 0 0 2.2 KB
#5595 EnrichedLog net472 4.7μs 2.41ns 9.34ns 0.32 0 0 2.02 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.36μs 0.904ns 3.5ns 0.0156 0 0 1.14 KB
master SendReceive netcoreapp3.1 1.68μs 0.742ns 2.78ns 0.0151 0 0 1.14 KB
master SendReceive net472 2.17μs 1.43ns 5.52ns 0.183 0 0 1.16 KB
#5595 SendReceive net6.0 1.29μs 0.61ns 2.36ns 0.0161 0 0 1.14 KB
#5595 SendReceive netcoreapp3.1 1.74μs 3.86ns 15ns 0.0156 0 0 1.14 KB
#5595 SendReceive net472 2.07μs 1.04ns 4.02ns 0.183 0 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.76μs 1.21ns 4.52ns 0.0222 0 0 1.6 KB
master EnrichedLog netcoreapp3.1 4.05μs 1.46ns 5.46ns 0.0226 0 0 1.65 KB
master EnrichedLog net472 4.46μs 2.82ns 10.5ns 0.324 0 0 2.04 KB
#5595 EnrichedLog net6.0 2.71μs 3.41ns 13.2ns 0.0214 0 0 1.6 KB
#5595 EnrichedLog netcoreapp3.1 3.8μs 1.84ns 7.13ns 0.0209 0 0 1.65 KB
#5595 EnrichedLog net472 4.46μs 3.46ns 13.4ns 0.323 0 0 2.04 KB
Benchmarks.Trace.SpanBenchmark - Slower :warning: Same allocations :heavy_check_mark:

Slower :warning: in #5595

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472 1.241 568.71 705.79
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑netcoreapp3.1 1.136 568.85 646.42

Faster :tada: in #5595

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0 1.135 536.36 472.44

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 394ns 0.12ns 0.464ns 0.00811 0 0 576 B
master StartFinishSpan netcoreapp3.1 569ns 0.275ns 1.07ns 0.00758 0 0 576 B
master StartFinishSpan net472 569ns 0.204ns 0.791ns 0.0917 0 0 578 B
master StartFinishScope net6.0 536ns 0.219ns 0.849ns 0.00965 0 0 696 B
master StartFinishScope netcoreapp3.1 650ns 0.438ns 1.69ns 0.00933 0 0 696 B
master StartFinishScope net472 832ns 0.616ns 2.39ns 0.104 0 0 658 B
#5595 StartFinishSpan net6.0 403ns 0.195ns 0.755ns 0.00816 0 0 576 B
#5595 StartFinishSpan netcoreapp3.1 646ns 0.334ns 1.25ns 0.00757 0 0 576 B
#5595 StartFinishSpan net472 707ns 1.11ns 4.31ns 0.0918 0 0 578 B
#5595 StartFinishScope net6.0 472ns 0.23ns 0.891ns 0.00989 0 0 696 B
#5595 StartFinishScope netcoreapp3.1 721ns 0.271ns 1.02ns 0.00942 0 0 696 B
#5595 StartFinishScope net472 901ns 0.456ns 1.76ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 634ns 0.458ns 1.77ns 0.00968 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 850ns 0.405ns 1.46ns 0.00929 0 0 696 B
master RunOnMethodBegin net472 1.03μs 0.364ns 1.36ns 0.104 0 0 658 B
#5595 RunOnMethodBegin net6.0 653ns 0.243ns 0.875ns 0.00984 0 0 696 B
#5595 RunOnMethodBegin netcoreapp3.1 852ns 0.432ns 1.67ns 0.00937 0 0 696 B
#5595 RunOnMethodBegin net472 1.03μs 0.7ns 2.71ns 0.105 0 0 658 B

andrewlock avatar May 22 '24 12:05 andrewlock

Benchmarks Report for appsec :snail:

Benchmarks for #5595 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.888
  • 2 benchmarks have fewer allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.Asm.AppSecBodyBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net6.0 72.2μs 139ns 537ns 0.0722 0 0 6.01 KB
master AllCycleSimpleBody netcoreapp3.1 61.8μs 104ns 401ns 0.0922 0 0 6.95 KB
master AllCycleSimpleBody net472 47.6μs 39.8ns 154ns 1.32 0 0 8.34 KB
master AllCycleMoreComplexBody net6.0 77.2μs 109ns 423ns 0.116 0 0 9.51 KB
master AllCycleMoreComplexBody netcoreapp3.1 68.8μs 88.5ns 343ns 0.138 0 0 10.37 KB
master AllCycleMoreComplexBody net472 54.4μs 42.9ns 166ns 1.88 0.0272 0 11.85 KB
master ObjectExtractorSimpleBody net6.0 147ns 0.154ns 0.595ns 0.00392 0 0 280 B
master ObjectExtractorSimpleBody netcoreapp3.1 198ns 0.224ns 0.775ns 0.00378 0 0 272 B
master ObjectExtractorSimpleBody net472 172ns 0.23ns 0.889ns 0.0446 0 0 281 B
master ObjectExtractorMoreComplexBody net6.0 3μs 1.37ns 5.11ns 0.053 0 0 3.78 KB
master ObjectExtractorMoreComplexBody netcoreapp3.1 4.09μs 1.88ns 7.27ns 0.0509 0 0 3.69 KB
master ObjectExtractorMoreComplexBody net472 3.75μs 2.79ns 10.8ns 0.603 0.00564 0 3.8 KB
#5595 AllCycleSimpleBody net6.0 72.2μs 122ns 473ns 0.0721 0 0 6.01 KB
#5595 AllCycleSimpleBody netcoreapp3.1 61.9μs 77.5ns 300ns 0.0937 0 0 6.95 KB
#5595 AllCycleSimpleBody net472 47.1μs 25.8ns 93.1ns 1.31 0 0 8.33 KB
#5595 AllCycleMoreComplexBody net6.0 77.2μs 67.2ns 251ns 0.116 0 0 9.51 KB
#5595 AllCycleMoreComplexBody netcoreapp3.1 68.9μs 99.3ns 372ns 0.137 0 0 10.37 KB
#5595 AllCycleMoreComplexBody net472 54.4μs 112ns 432ns 1.88 0.0273 0 11.85 KB
#5595 ObjectExtractorSimpleBody net6.0 140ns 0.104ns 0.402ns 0.00394 0 0 280 B
#5595 ObjectExtractorSimpleBody netcoreapp3.1 219ns 0.172ns 0.644ns 0.00366 0 0 272 B
#5595 ObjectExtractorSimpleBody net472 170ns 0.153ns 0.571ns 0.0446 0 0 281 B
#5595 ObjectExtractorMoreComplexBody net6.0 3.03μs 1.26ns 4.71ns 0.0531 0 0 3.78 KB
#5595 ObjectExtractorMoreComplexBody netcoreapp3.1 3.95μs 1.86ns 6.98ns 0.0493 0 0 3.69 KB
#5595 ObjectExtractorMoreComplexBody net472 3.86μs 2.72ns 9.79ns 0.602 0.00577 0 3.8 KB
Benchmarks.Trace.Asm.AppSecEncoderBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EncodeArgs net6.0 38.2μs 12.9ns 48.1ns 0.457 0 0 32.4 KB
master EncodeArgs netcoreapp3.1 54.1μs 23.8ns 88.9ns 0.432 0 0 32.4 KB
master EncodeArgs net472 67.7μs 25.7ns 96.3ns 5.13 0.0675 0 32.5 KB
master EncodeLegacyArgs net6.0 75.4μs 89.6ns 347ns 0 0 0 2.14 KB
master EncodeLegacyArgs netcoreapp3.1 105μs 111ns 431ns 0 0 0 2.14 KB
master EncodeLegacyArgs net472 157μs 113ns 437ns 0.313 0 0 2.15 KB
#5595 EncodeArgs net6.0 38.6μs 15.9ns 59.6ns 0.443 0 0 32.4 KB
#5595 EncodeArgs netcoreapp3.1 55.1μs 148ns 572ns 0.433 0 0 32.4 KB
#5595 EncodeArgs net472 67.1μs 31.1ns 116ns 5.14 0.0668 0 32.5 KB
#5595 EncodeLegacyArgs net6.0 69.9μs 131ns 508ns 0 0 0 2.14 KB
#5595 EncodeLegacyArgs netcoreapp3.1 104μs 196ns 758ns 0 0 0 2.14 KB
#5595 EncodeLegacyArgs net472 159μs 112ns 435ns 0.314 0 0 2.15 KB
Benchmarks.Trace.Asm.AppSecWafBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunWafRealisticBenchmark net6.0 181μs 140ns 543ns 0 0 0 2.42 KB
master RunWafRealisticBenchmark netcoreapp3.1 194μs 176ns 682ns 0 0 0 2.37 KB
master RunWafRealisticBenchmark net472 208μs 29.8ns 115ns 0.31 0 0 2.43 KB
master RunWafRealisticBenchmarkWithAttack net6.0 122μs 49.3ns 191ns 0 0 0 1.46 KB
master RunWafRealisticBenchmarkWithAttack netcoreapp3.1 129μs 156ns 583ns 0 0 0 1.45 KB
master RunWafRealisticBenchmarkWithAttack net472 139μs 93.8ns 363ns 0.211 0 0 1.48 KB
#5595 RunWafRealisticBenchmark net6.0 182μs 101ns 393ns 0 0 0 2.42 KB
#5595 RunWafRealisticBenchmark netcoreapp3.1 194μs 163ns 630ns 0 0 0 2.37 KB
#5595 RunWafRealisticBenchmark net472 210μs 331ns 1.28μs 0.314 0 0 2.43 KB
#5595 RunWafRealisticBenchmarkWithAttack net6.0 121μs 78.9ns 305ns 0 0 0 1.46 KB
#5595 RunWafRealisticBenchmarkWithAttack netcoreapp3.1 130μs 152ns 589ns 0 0 0 1.45 KB
#5595 RunWafRealisticBenchmarkWithAttack net472 141μs 51.6ns 200ns 0.21 0 0 1.48 KB
Benchmarks.Trace.Iast.StringAspectsBenchmark - Faster :tada: Fewer allocations :tada:

Faster :tada: in #5595

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1 1.888 613,800.00 325,150.00

Fewer allocations :tada: in #5595

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatBenchmark‑net472 61.6 KB 59.35 KB -2.25 KB -3.65%
Benchmarks.Trace.Iast.StringAspectsBenchmark.StringConcatAspectBenchmark‑netcoreapp3.1 326 KB 253.83 KB -72.17 KB -22.14%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StringConcatBenchmark net6.0 60.5μs 1μs 9.99μs 0 0 0 43.44 KB
master StringConcatBenchmark netcoreapp3.1 61.6μs 763ns 7.4μs 0 0 0 42.64 KB
master StringConcatBenchmark net472 38.2μs 97ns 350ns 0 0 0 61.6 KB
master StringConcatAspectBenchmark net6.0 310μs 1.69μs 10.6μs 0 0 0 255.68 KB
master StringConcatAspectBenchmark netcoreapp3.1 618μs 3.17μs 14.5μs 0 0 0 326 KB
master StringConcatAspectBenchmark net472 302μs 5.9μs 56.6μs 0 0 0 278.53 KB
#5595 StringConcatBenchmark net6.0 61.5μs 782ns 7.82μs 0 0 0 43.44 KB
#5595 StringConcatBenchmark netcoreapp3.1 63.3μs 821ns 8.13μs 0 0 0 42.64 KB
#5595 StringConcatBenchmark net472 38.5μs 73.8ns 266ns 0 0 0 59.35 KB
#5595 StringConcatAspectBenchmark net6.0 319μs 1.28μs 4.42μs 0 0 0 255.99 KB
#5595 StringConcatAspectBenchmark netcoreapp3.1 329μs 1.74μs 11.8μs 0 0 0 253.83 KB
#5595 StringConcatAspectBenchmark net472 306μs 5.84μs 56.4μs 0 0 0 278.53 KB

andrewlock avatar Jun 05 '24 18:06 andrewlock

Datadog Report

Branch report: andrew/ci/glibc-musl-packaging Commit report: d424af1 Test service: dd-trace-dotnet

:x: 1 Failed (0 Known Flaky), 426729 Passed, 3375 Skipped, 34h 40m 59.71s Total Time :hourglass: 1 Performance Regression

:x: Failed Tests (1)

  • CheckSmokeForOldWayToStackWalk - Datadog.Profiler.SmokeTests.ExceptionGeneratorTest - Details

    Expand for error 
    he test Samples.ExceptionGenerator is running for too long or was lost

:hourglass: Performance Regressions vs Default Branch (1)

  • Baseline_garbagecollections - scenarios 1.21s (+84.78ms, +8%) - Details

datadog-ddstaging[bot] avatar Jul 04 '24 09:07 datadog-ddstaging[bot]