dd-trace-dotnet icon indicating copy to clipboard operation
dd-trace-dotnet copied to clipboard
verified publisher icon DataDog

Output samples to a single top-level "artifacts" folder

Open andrewlock opened this issue 1 year ago • 4 comments

Summary of changes

  • Use the .NET 8 SDK feature "artifacts layout"
  • Apply to all samples so they build to a single top-level artifacts folder

Reason for change

This is the first step of various CI improvements I'd like to make. By building all the samples to a single location this should

  • Make it easier to "find" the samples in the test code (currently it's still confusing, but that will be easier to tidy up later)
  • Allow building all the samples once for a pipeline, reducing the time in the critical path (i.e. faster CI)
  • If there's no changes in sample apps, re-use the sample build from the master branch

These are mostly academic benefits right now, but I'm working on PoCs for them.

Implementation details

  • Set UseArtifactsOutput so the .NET SDK defaults to placing the bin, obj, and publish folders inside a top-level artifacts folder
  • Customize the pivot when we're building multiple package versions, to include the package version in the pivot
  • Workarounds for various annoying projects, e.g. NServiceBus, XUnit, AWS serverless tests
  • Move the build_data directory to artifacts/build_data

Test coverage

If it all builds and runs successfully, we're good at this stage!

Other details

Later on we can move everything to use the artifacts layout, but one step at a time!

andrewlock avatar Jun 26 '24 14:06 andrewlock

Datadog Report

Branch report: andrew/ci/use-artifacts-output-location Commit report: a507797 Test service: dd-trace-dotnet

:white_check_mark: 0 Failed, 350839 Passed, 2269 Skipped, 23h 15m 27.2s Total Time

datadog-ddstaging[bot] avatar Jun 26 '24 14:06 datadog-ddstaging[bot]

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    master - mean (74ms)  : 64, 85
     .   : milestone, 74,

    section CallTarget+Inlining+NGEN
    master - mean (1,110ms)  : 1095, 1126
     .   : milestone, 1110,


gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    master - mean (109ms)  : 106, 113
     .   : milestone, 109,

    section CallTarget+Inlining+NGEN
    master - mean (807ms)  : 790, 825
     .   : milestone, 807,


gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    master - mean (93ms)  : 88, 98
     .   : milestone, 93,

    section CallTarget+Inlining+NGEN
    master - mean (756ms)  : 736, 775
     .   : milestone, 756,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    master - mean (190ms)  : 187, 193
     .   : milestone, 190,

    section CallTarget+Inlining+NGEN
    master - mean (1,197ms)  : 1177, 1218
     .   : milestone, 1197,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    master - mean (275ms)  : 268, 282
     .   : milestone, 275,

    section CallTarget+Inlining+NGEN
    master - mean (976ms)  : 954, 997
     .   : milestone, 976,


gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    master - mean (263ms)  : 259, 267
     .   : milestone, 263,

    section CallTarget+Inlining+NGEN
    master - mean (961ms)  : 941, 981
     .   : milestone, 961,

andrewlock avatar Jun 26 '24 15:06 andrewlock

Benchmarks Report for tracer :snail:

Benchmarks for #5744 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.160
  • All benchmarks have the same allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 7.78μs 42.7ns 245ns 0.019 0.00761 0 5.42 KB
master StartStopWithChild netcoreapp3.1 9.96μs 56.5ns 400ns 0.0147 0.0049 0 5.62 KB
master StartStopWithChild net472 16.2μs 78ns 312ns 1.01 0.287 0.0878 6.07 KB
#5744 StartStopWithChild net6.0 7.63μs 40.5ns 289ns 0.0156 0.0078 0 5.42 KB
#5744 StartStopWithChild netcoreapp3.1 9.84μs 51.1ns 240ns 0.0186 0.00932 0 5.62 KB
#5744 StartStopWithChild net472 16.1μs 34.1ns 132ns 1 0.286 0.0898 6.06 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 476μs 290ns 1.12μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 623μs 357ns 1.38μs 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 842μs 404ns 1.51μs 0.425 0 0 3.3 KB
#5744 WriteAndFlushEnrichedTraces net6.0 477μs 246ns 952ns 0 0 0 2.7 KB
#5744 WriteAndFlushEnrichedTraces netcoreapp3.1 630μs 208ns 780ns 0 0 0 2.7 KB
#5744 WriteAndFlushEnrichedTraces net472 847μs 527ns 1.97μs 0.422 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 191μs 1.02μs 7.45μs 0.182 0 0 18.45 KB
master SendRequest netcoreapp3.1 212μs 1.15μs 6.87μs 0.213 0 0 20.61 KB
master SendRequest net472 0.000489ns 0.000276ns 0.000955ns 0 0 0 0 b
#5744 SendRequest net6.0 189μs 810ns 2.81μs 0.214 0 0 18.45 KB
#5744 SendRequest netcoreapp3.1 216μs 1.19μs 6.91μs 0.224 0 0 20.61 KB
#5744 SendRequest net472 0.000528ns 0.000212ns 0.000794ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 573μs 2.71μs 11.5μs 0.556 0 0 41.76 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 684μs 3.68μs 21.5μs 0.343 0 0 41.71 KB
master WriteAndFlushEnrichedTraces net472 855μs 3.92μs 15.2μs 8.39 2.52 0.419 53.27 KB
#5744 WriteAndFlushEnrichedTraces net6.0 582μs 3μs 14.4μs 0.532 0 0 41.74 KB
#5744 WriteAndFlushEnrichedTraces netcoreapp3.1 678μs 2.83μs 10.2μs 0.336 0 0 41.73 KB
#5744 WriteAndFlushEnrichedTraces net472 870μs 4.05μs 15.2μs 8.5 2.55 0.425 53.31 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.3μs 1.3ns 4.85ns 0.0143 0 0 1.02 KB
master ExecuteNonQuery netcoreapp3.1 1.64μs 1.28ns 4.95ns 0.0138 0 0 1.02 KB
master ExecuteNonQuery net472 2.05μs 2ns 7.76ns 0.156 0 0 987 B
#5744 ExecuteNonQuery net6.0 1.22μs 0.71ns 2.75ns 0.0146 0 0 1.02 KB
#5744 ExecuteNonQuery netcoreapp3.1 1.77μs 1.18ns 4.55ns 0.0133 0 0 1.02 KB
#5744 ExecuteNonQuery net472 2.02μs 2.05ns 7.95ns 0.157 0 0 987 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.2μs 0.529ns 1.98ns 0.0139 0 0 976 B
master CallElasticsearch netcoreapp3.1 1.58μs 0.567ns 2.12ns 0.0135 0 0 976 B
master CallElasticsearch net472 2.46μs 2.99ns 11.6ns 0.158 0 0 995 B
master CallElasticsearchAsync net6.0 1.32μs 0.909ns 3.52ns 0.0132 0 0 952 B
master CallElasticsearchAsync netcoreapp3.1 1.66μs 0.686ns 2.57ns 0.0142 0 0 1.02 KB
master CallElasticsearchAsync net472 2.72μs 1.02ns 3.68ns 0.166 0 0 1.05 KB
#5744 CallElasticsearch net6.0 1.23μs 0.942ns 3.52ns 0.0135 0 0 976 B
#5744 CallElasticsearch netcoreapp3.1 1.55μs 0.971ns 3.36ns 0.0132 0 0 976 B
#5744 CallElasticsearch net472 2.45μs 0.661ns 2.47ns 0.158 0 0 995 B
#5744 CallElasticsearchAsync net6.0 1.36μs 0.929ns 3.47ns 0.013 0 0 952 B
#5744 CallElasticsearchAsync netcoreapp3.1 1.71μs 0.736ns 2.85ns 0.0135 0 0 1.02 KB
#5744 CallElasticsearchAsync net472 2.58μs 1.9ns 7.36ns 0.166 0 0 1.05 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.2μs 0.777ns 2.91ns 0.0133 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.55μs 0.996ns 3.86ns 0.0133 0 0 952 B
master ExecuteAsync net472 1.83μs 3.97ns 15.4ns 0.145 0 0 915 B
#5744 ExecuteAsync net6.0 1.29μs 0.898ns 3.36ns 0.0135 0 0 952 B
#5744 ExecuteAsync netcoreapp3.1 1.59μs 0.381ns 1.43ns 0.0129 0 0 952 B
#5744 ExecuteAsync net472 1.76μs 1.04ns 4.02ns 0.145 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.17μs 1.39ns 5.19ns 0.0313 0 0 2.22 KB
master SendAsync netcoreapp3.1 5.15μs 2.78ns 10.8ns 0.036 0 0 2.76 KB
master SendAsync net472 7.88μs 1.9ns 7.1ns 0.498 0 0 3.15 KB
#5744 SendAsync net6.0 4.12μs 2ns 7.76ns 0.0309 0 0 2.22 KB
#5744 SendAsync netcoreapp3.1 5.1μs 1.89ns 6.82ns 0.0357 0 0 2.76 KB
#5744 SendAsync net472 7.87μs 1.13ns 4.36ns 0.499 0 0 3.15 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.56μs 0.735ns 2.85ns 0.0233 0 0 1.64 KB
master EnrichedLog netcoreapp3.1 2.36μs 0.77ns 2.88ns 0.0221 0 0 1.64 KB
master EnrichedLog net472 2.65μs 1.19ns 4.62ns 0.249 0 0 1.57 KB
#5744 EnrichedLog net6.0 1.7μs 0.422ns 1.58ns 0.0229 0 0 1.64 KB
#5744 EnrichedLog netcoreapp3.1 2.27μs 1.16ns 4.5ns 0.0215 0 0 1.64 KB
#5744 EnrichedLog net472 2.61μs 0.891ns 3.45ns 0.249 0 0 1.57 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 117μs 212ns 792ns 0.0584 0 0 4.28 KB
master EnrichedLog netcoreapp3.1 120μs 262ns 1.01μs 0 0 0 4.28 KB
master EnrichedLog net472 150μs 132ns 510ns 0.67 0.223 0 4.46 KB
#5744 EnrichedLog net6.0 119μs 208ns 805ns 0.06 0 0 4.28 KB
#5744 EnrichedLog netcoreapp3.1 121μs 253ns 981ns 0 0 0 4.28 KB
#5744 EnrichedLog net472 153μs 297ns 1.15μs 0.683 0.228 0 4.46 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.06μs 0.983ns 3.81ns 0.0302 0 0 2.2 KB
master EnrichedLog netcoreapp3.1 4.24μs 1.89ns 7.31ns 0.0298 0 0 2.2 KB
master EnrichedLog net472 4.89μs 1.72ns 6.21ns 0.32 0 0 2.02 KB
#5744 EnrichedLog net6.0 2.99μs 0.76ns 2.94ns 0.03 0 0 2.2 KB
#5744 EnrichedLog netcoreapp3.1 4.17μs 1.75ns 6.77ns 0.0279 0 0 2.2 KB
#5744 EnrichedLog net472 4.75μs 0.881ns 3.3ns 0.32 0 0 2.02 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.3μs 0.355ns 1.38ns 0.0161 0 0 1.14 KB
master SendReceive netcoreapp3.1 1.79μs 1.41ns 5.47ns 0.0152 0 0 1.14 KB
master SendReceive net472 2.06μs 0.812ns 3.04ns 0.183 0 0 1.16 KB
#5744 SendReceive net6.0 1.41μs 0.511ns 1.98ns 0.0161 0 0 1.14 KB
#5744 SendReceive netcoreapp3.1 1.78μs 0.967ns 3.74ns 0.0153 0 0 1.14 KB
#5744 SendReceive net472 2.14μs 0.669ns 2.5ns 0.183 0.00107 0 1.16 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.7μs 1.02ns 3.96ns 0.0217 0 0 1.6 KB
master EnrichedLog netcoreapp3.1 3.85μs 1.92ns 7.17ns 0.0212 0 0 1.65 KB
master EnrichedLog net472 4.31μs 2.53ns 9.79ns 0.323 0 0 2.04 KB
#5744 EnrichedLog net6.0 2.73μs 1.02ns 3.83ns 0.0219 0 0 1.6 KB
#5744 EnrichedLog netcoreapp3.1 3.93μs 2.32ns 8.66ns 0.0217 0 0 1.65 KB
#5744 EnrichedLog net472 4.47μs 1.28ns 4.77ns 0.324 0 0 2.04 KB
Benchmarks.Trace.SpanBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 397ns 0.324ns 1.25ns 0.00814 0 0 576 B
master StartFinishSpan netcoreapp3.1 535ns 0.435ns 1.68ns 0.00774 0 0 576 B
master StartFinishSpan net472 576ns 0.943ns 3.53ns 0.0917 0 0 578 B
master StartFinishScope net6.0 503ns 0.438ns 1.7ns 0.00973 0 0 696 B
master StartFinishScope netcoreapp3.1 729ns 0.877ns 3.39ns 0.00932 0 0 696 B
master StartFinishScope net472 844ns 1.3ns 5.03ns 0.104 0 0 658 B
#5744 StartFinishSpan net6.0 427ns 0.29ns 1.12ns 0.00807 0 0 576 B
#5744 StartFinishSpan netcoreapp3.1 571ns 0.615ns 2.3ns 0.00779 0 0 576 B
#5744 StartFinishSpan net472 634ns 0.566ns 2.19ns 0.0917 0 0 578 B
#5744 StartFinishScope net6.0 537ns 0.523ns 2.03ns 0.00964 0 0 696 B
#5744 StartFinishScope netcoreapp3.1 765ns 0.962ns 3.73ns 0.00946 0 0 696 B
#5744 StartFinishScope net472 924ns 1.6ns 6.21ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Faster :tada: Same allocations :heavy_check_mark:

Faster :tada: in #5744

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0 1.160 684.95 590.50

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 684ns 0.641ns 2.48ns 0.00994 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 914ns 0.535ns 2.07ns 0.00959 0 0 696 B
master RunOnMethodBegin net472 1.06μs 1.31ns 5.05ns 0.104 0 0 658 B
#5744 RunOnMethodBegin net6.0 590ns 0.444ns 1.72ns 0.00978 0 0 696 B
#5744 RunOnMethodBegin netcoreapp3.1 862ns 0.58ns 2.25ns 0.00928 0 0 696 B
#5744 RunOnMethodBegin net472 1.08μs 1.07ns 3.86ns 0.104 0 0 658 B

andrewlock avatar Jun 26 '24 15:06 andrewlock

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5744) (11.545M)   : 0, 11545290
    master (11.593M)   : 0, 11593369
    benchmarks/2.9.0 (11.542M)   : 0, 11542126

    section Automatic
    This PR (5744) (7.856M)   : 0, 7856497
    master (7.827M)   : 0, 7827212
    benchmarks/2.9.0 (8.263M)   : 0, 8262905

    section Trace stats
    master (8.118M)   : 0, 8118390

    section Manual
    This PR (5744) (10.012M)   : 0, 10012358
    master (9.920M)   : 0, 9920204

    section Manual + Automatic
    This PR (5744) (7.380M)   : 0, 7379888
    master (7.358M)   : 0, 7357979

    section Version Conflict
    master (6.635M)   : 0, 6634804


gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5744) (9.703M)   : 0, 9703003
    master (9.689M)   : 0, 9688586
    benchmarks/2.9.0 (9.596M)   : 0, 9596140

    section Automatic
    This PR (5744) (6.648M)   : 0, 6647521
    master (6.298M)   : 0, 6298066

    section Trace stats
    master (6.862M)   : 0, 6861645

    section Manual
    This PR (5744) (8.251M)   : 0, 8251316
    master (8.028M)   : 0, 8028063

    section Manual + Automatic
    This PR (5744) (6.188M)   : 0, 6187886
    master (6.139M)   : 0, 6139377

    section Version Conflict
    master (5.667M)   : 0, 5667497


gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5744) (9.932M)   : 0, 9932293
    master (10.241M)   : 0, 10241408
    benchmarks/2.9.0 (10.213M)   : 0, 10213239

    section Automatic
    This PR (5744) (7.158M)   : 0, 7158055
    master (7.186M)   : 0, 7186302
    benchmarks/2.9.0 (7.482M)   : 0, 7482023

    section Trace stats
    master (7.568M)   : 0, 7568098

    section Manual
    This PR (5744) (8.883M)   : 0, 8883396
    master (9.213M)   : 0, 9213352

    section Manual + Automatic
    This PR (5744) (6.840M)   : 0, 6839984
    master (7.004M)   : 0, 7004280

    section Version Conflict
    master (6.289M)   : 0, 6288694

andrewlock avatar Jun 26 '24 21:06 andrewlock