dd-trace-dotnet icon indicating copy to clipboard operation
dd-trace-dotnet copied to clipboard
verified publisher icon DataDog

Run integrations tests based on modification in PRs

Open gleocadie opened this issue 1 year ago • 5 comments

Summary of changes

Run integrations tests job based on modification(s) in PRs.

Reason for change

Today when a modification is done in the Profiler, tracer integration tests run. When modification are done in the Tracer, profiler integration tests subset is not run (with CodeHotspot, End point aggregation...).

The goal of this PR is to run profiler integration tests on Profiler PRs (only), run the tracer integration tests + a subset of profiler integration tests (Code Hotspot...). This may speed up the CI for profiler PRs.

Implementation details

  • Detect at runtime if the changes are related to the profiler and/or the tracer
  • For profiler integration tests, pass a flag to update the test filter (done in this PR)

Test coverage

Other details

gleocadie avatar Apr 25 '24 11:04 gleocadie

Datadog Report

Branch report: gleocadie/seggrate-profiler-tests-to-run-on-a-tracer-pr Commit report: 13aacdf Test service: dd-trace-dotnet

:white_check_mark: 0 Failed, 308454 Passed, 1068 Skipped, 35m 51.31s Wall Time

datadog-ddstaging[bot] avatar Apr 25 '24 12:04 datadog-ddstaging[bot]

Execution-Time Benchmarks Report :stopwatch:

Execution-time results for samples comparing the following branches/commits:

Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:

  • Welch test with statistical test for significance of 5%
  • Only results indicating a difference greater than 5% and 5 ms are considered.

Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.

Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).

gantt
    title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5499) - mean (72ms)  : 66, 79
     .   : milestone, 72,
    master - mean (71ms)  : 67, 76
     .   : milestone, 71,

    section CallTarget+Inlining+NGEN
    This PR (5499) - mean (1,022ms)  : 997, 1047
     .   : milestone, 1022,
    master - mean (1,021ms)  : 1001, 1041
     .   : milestone, 1021,


gantt
    title Execution time (ms) FakeDbCommand (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5499) - mean (110ms)  : 107, 112
     .   : milestone, 110,
    master - mean (110ms)  : 107, 114
     .   : milestone, 110,

    section CallTarget+Inlining+NGEN
    This PR (5499) - mean (733ms)  : 712, 754
     .   : milestone, 733,
    master - mean (736ms)  : 711, 762
     .   : milestone, 736,


gantt
    title Execution time (ms) FakeDbCommand (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5499) - mean (93ms)  : 90, 97
     .   : milestone, 93,
    master - mean (94ms)  : 91, 97
     .   : milestone, 94,

    section CallTarget+Inlining+NGEN
    This PR (5499) - mean (689ms)  : 666, 712
     .   : milestone, 689,
    master - mean (689ms)  : 668, 710
     .   : milestone, 689,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5499) - mean (192ms)  : 189, 196
     .   : milestone, 192,
    master - mean (190ms)  : 186, 194
     .   : milestone, 190,

    section CallTarget+Inlining+NGEN
    This PR (5499) - mean (1,099ms)  : 1075, 1123
     .   : milestone, 1099,
    master - mean (1,098ms)  : 1069, 1126
     .   : milestone, 1098,


gantt
    title Execution time (ms) HttpMessageHandler (.NET Core 3.1) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5499) - mean (276ms)  : 271, 281
     .   : milestone, 276,
    master - mean (276ms)  : 272, 281
     .   : milestone, 276,

    section CallTarget+Inlining+NGEN
    This PR (5499) - mean (889ms)  : 868, 910
     .   : milestone, 889,
    master - mean (889ms)  : 865, 914
     .   : milestone, 889,


gantt
    title Execution time (ms) HttpMessageHandler (.NET 6) 
    dateFormat  X
    axisFormat %s
    todayMarker off
    section Baseline
    This PR (5499) - mean (264ms)  : 260, 269
     .   : milestone, 264,
    master - mean (264ms)  : 260, 268
     .   : milestone, 264,

    section CallTarget+Inlining+NGEN
    This PR (5499) - mean (870ms)  : 841, 900
     .   : milestone, 870,
    master - mean (870ms)  : 847, 893
     .   : milestone, 870,

andrewlock avatar Apr 25 '24 12:04 andrewlock

Benchmarks Report for tracer :snail:

Benchmarks for #5499 compared to master:

  • 2 benchmarks are faster, with geometric mean 1.157
  • 1 benchmarks are slower, with geometric mean 1.276
  • 1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.ActivityBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartStopWithChild net6.0 8.84μs 50.2ns 355ns 0.0255 0.0128 0 7.56 KB
master StartStopWithChild netcoreapp3.1 10.8μs 59.5ns 336ns 0.0265 0.0106 0 7.65 KB
master StartStopWithChild net472 17.1μs 39.1ns 152ns 1.35 0.348 0.0934 8.09 KB
#5499 StartStopWithChild net6.0 8.68μs 47.8ns 287ns 0.0267 0.0133 0 7.55 KB
#5499 StartStopWithChild netcoreapp3.1 10.8μs 59.7ns 391ns 0.0312 0.0104 0 7.65 KB
#5499 StartStopWithChild net472 16.9μs 35.9ns 139ns 1.37 0.375 0.0917 8.1 KB
Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 477μs 234ns 876ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 623μs 126ns 471ns 0 0 0 2.7 KB
master WriteAndFlushEnrichedTraces net472 829μs 168ns 605ns 0.414 0 0 3.3 KB
#5499 WriteAndFlushEnrichedTraces net6.0 480μs 174ns 652ns 0 0 0 2.7 KB
#5499 WriteAndFlushEnrichedTraces netcoreapp3.1 633μs 179ns 671ns 0 0 0 2.7 KB
#5499 WriteAndFlushEnrichedTraces net472 826μs 238ns 889ns 0.414 0 0 3.3 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net6.0 170μs 189ns 733ns 0.254 0 0 18.49 KB
master SendRequest netcoreapp3.1 195μs 320ns 1.24μs 0.193 0 0 20.65 KB
master SendRequest net472 0.000235ns 0.000132ns 0.000476ns 0 0 0 0 b
#5499 SendRequest net6.0 172μs 227ns 880ns 0.253 0 0 18.49 KB
#5499 SendRequest netcoreapp3.1 193μs 294ns 1.14μs 0.193 0 0 20.65 KB
#5499 SendRequest net472 0.000573ns 0.000207ns 0.0008ns 0 0 0 0 b
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark - Same speed :heavy_check_mark: More allocations :warning:

More allocations :warning: in #5499

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.CIVisibilityProtocolWriterBenchmark.WriteAndFlushEnrichedTraces‑netcoreapp3.1 41.79 KB 42.03 KB 239 B 0.57%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net6.0 557μs 1.9μs 7.34μs 0.566 0 0 41.6 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 663μs 837ns 3.24μs 0.331 0 0 41.79 KB
master WriteAndFlushEnrichedTraces net472 900μs 4.12μs 16.5μs 8.3 2.62 0.437 53.26 KB
#5499 WriteAndFlushEnrichedTraces net6.0 562μs 481ns 1.67μs 0.573 0 0 41.67 KB
#5499 WriteAndFlushEnrichedTraces netcoreapp3.1 657μs 1.88μs 7.27μs 0.329 0 0 42.03 KB
#5499 WriteAndFlushEnrichedTraces net472 846μs 3.4μs 12.7μs 8.28 2.48 0.414 53.27 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net6.0 1.24μs 0.631ns 2.44ns 0.0112 0 0 808 B
master ExecuteNonQuery netcoreapp3.1 1.51μs 0.637ns 2.47ns 0.0108 0 0 808 B
master ExecuteNonQuery net472 1.77μs 0.547ns 2.12ns 0.122 0 0 770 B
#5499 ExecuteNonQuery net6.0 1.22μs 0.483ns 1.87ns 0.011 0 0 808 B
#5499 ExecuteNonQuery netcoreapp3.1 1.59μs 0.384ns 1.44ns 0.0103 0 0 808 B
#5499 ExecuteNonQuery net472 1.8μs 0.767ns 2.87ns 0.122 0 0 770 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net6.0 1.26μs 1.23ns 4.61ns 0.0147 0 0 1.03 KB
master CallElasticsearch netcoreapp3.1 1.68μs 0.942ns 3.53ns 0.0133 0 0 1.03 KB
master CallElasticsearch net472 2.51μs 1.36ns 5.26ns 0.165 0 0 1.04 KB
master CallElasticsearchAsync net6.0 1.45μs 1.06ns 4.1ns 0.0138 0 0 1.01 KB
master CallElasticsearchAsync netcoreapp3.1 1.76μs 0.811ns 3.03ns 0.0149 0 0 1.08 KB
master CallElasticsearchAsync net472 2.55μs 0.971ns 3.76ns 0.174 0 0 1.1 KB
#5499 CallElasticsearch net6.0 1.29μs 1.05ns 3.93ns 0.0144 0 0 1.03 KB
#5499 CallElasticsearch netcoreapp3.1 1.68μs 1.92ns 7.43ns 0.0135 0 0 1.03 KB
#5499 CallElasticsearch net472 2.65μs 0.952ns 3.56ns 0.165 0 0 1.04 KB
#5499 CallElasticsearchAsync net6.0 1.41μs 1.5ns 5.82ns 0.0141 0 0 1.01 KB
#5499 CallElasticsearchAsync netcoreapp3.1 1.78μs 0.943ns 3.53ns 0.0143 0 0 1.08 KB
#5499 CallElasticsearchAsync net472 2.62μs 0.492ns 1.77ns 0.174 0 0 1.1 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net6.0 1.45μs 0.885ns 3.31ns 0.0131 0 0 952 B
master ExecuteAsync netcoreapp3.1 1.74μs 3.98ns 14.9ns 0.0129 0 0 952 B
master ExecuteAsync net472 1.75μs 2.34ns 9.07ns 0.145 0 0 915 B
#5499 ExecuteAsync net6.0 1.43μs 0.66ns 2.47ns 0.013 0 0 952 B
#5499 ExecuteAsync netcoreapp3.1 1.68μs 0.607ns 2.35ns 0.0128 0 0 952 B
#5499 ExecuteAsync net472 1.75μs 0.572ns 2.22ns 0.145 0 0 915 B
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net6.0 4.08μs 1.39ns 5.18ns 0.0308 0 0 2.27 KB
master SendAsync netcoreapp3.1 5.15μs 4.14ns 14.3ns 0.0385 0 0 2.81 KB
master SendAsync net472 7.83μs 2.92ns 11.3ns 0.504 0 0 3.18 KB
#5499 SendAsync net6.0 4.23μs 4.2ns 15.7ns 0.0316 0 0 2.27 KB
#5499 SendAsync netcoreapp3.1 5.18μs 1.6ns 6ns 0.0364 0 0 2.81 KB
#5499 SendAsync net472 7.79μs 2.26ns 8.45ns 0.503 0 0 3.18 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 1.54μs 1.12ns 4.32ns 0.0239 0 0 1.7 KB
master EnrichedLog netcoreapp3.1 2.27μs 1.44ns 5.21ns 0.0227 0 0 1.7 KB
master EnrichedLog net472 2.77μs 3.92ns 15.2ns 0.256 0 0 1.62 KB
#5499 EnrichedLog net6.0 1.57μs 1.05ns 3.91ns 0.0236 0 0 1.7 KB
#5499 EnrichedLog netcoreapp3.1 2.24μs 1.54ns 5.75ns 0.0224 0 0 1.7 KB
#5499 EnrichedLog net472 2.68μs 1.8ns 6.5ns 0.257 0 0 1.62 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 115μs 230ns 859ns 0.0575 0 0 4.28 KB
master EnrichedLog netcoreapp3.1 121μs 202ns 781ns 0 0 0 4.28 KB
master EnrichedLog net472 150μs 192ns 743ns 0.673 0.224 0 4.46 KB
#5499 EnrichedLog net6.0 116μs 169ns 655ns 0.0581 0 0 4.28 KB
#5499 EnrichedLog netcoreapp3.1 121μs 235ns 910ns 0 0 0 4.28 KB
#5499 EnrichedLog net472 151μs 199ns 769ns 0.68 0.227 0 4.46 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 3.15μs 0.857ns 3.09ns 0.0315 0 0 2.25 KB
master EnrichedLog netcoreapp3.1 4.23μs 4.17ns 15.6ns 0.0295 0 0 2.25 KB
master EnrichedLog net472 4.84μs 2.24ns 8.66ns 0.328 0 0 2.07 KB
#5499 EnrichedLog net6.0 3.04μs 1.7ns 6.34ns 0.0319 0 0 2.25 KB
#5499 EnrichedLog netcoreapp3.1 4.19μs 2.03ns 7.86ns 0.0296 0 0 2.25 KB
#5499 EnrichedLog net472 4.96μs 1.59ns 6.15ns 0.327 0 0 2.07 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net6.0 1.46μs 1.19ns 4.6ns 0.0168 0 0 1.2 KB
master SendReceive netcoreapp3.1 1.84μs 1.35ns 5.06ns 0.0167 0 0 1.2 KB
master SendReceive net472 2.26μs 1.92ns 7.42ns 0.19 0 0 1.2 KB
#5499 SendReceive net6.0 1.47μs 0.904ns 3.5ns 0.0169 0 0 1.2 KB
#5499 SendReceive netcoreapp3.1 1.96μs 1.52ns 5.49ns 0.0167 0 0 1.2 KB
#5499 SendReceive net472 2.15μs 1.51ns 5.63ns 0.191 0 0 1.2 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net6.0 2.72μs 0.799ns 3.1ns 0.0222 0 0 1.6 KB
master EnrichedLog netcoreapp3.1 4.07μs 1.12ns 4.2ns 0.0208 0 0 1.65 KB
master EnrichedLog net472 4.34μs 0.9ns 3.37ns 0.322 0 0 2.04 KB
#5499 EnrichedLog net6.0 2.76μs 0.952ns 3.69ns 0.0221 0 0 1.6 KB
#5499 EnrichedLog netcoreapp3.1 4.01μs 1.47ns 5.29ns 0.022 0 0 1.65 KB
#5499 EnrichedLog net472 4.42μs 1.34ns 5.2ns 0.322 0 0 2.04 KB
Benchmarks.Trace.SpanBenchmark - Faster :tada: Same allocations :heavy_check_mark:

Faster :tada: in #5499

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishScope‑net6.0 1.162 698.56 601.02
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net6.0 1.151 562.65 488.91

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net6.0 563ns 0.189ns 0.733ns 0.00808 0 0 576 B
master StartFinishSpan netcoreapp3.1 681ns 3.48ns 15.2ns 0.00788 0 0 576 B
master StartFinishSpan net472 823ns 0.256ns 0.993ns 0.0914 0 0 578 B
master StartFinishScope net6.0 699ns 0.333ns 1.25ns 0.00984 0 0 696 B
master StartFinishScope netcoreapp3.1 878ns 0.269ns 1.01ns 0.00967 0 0 696 B
master StartFinishScope net472 1.03μs 0.684ns 2.65ns 0.105 0 0 658 B
#5499 StartFinishSpan net6.0 489ns 0.157ns 0.588ns 0.00805 0 0 576 B
#5499 StartFinishSpan netcoreapp3.1 736ns 2.02ns 7.83ns 0.00791 0 0 576 B
#5499 StartFinishSpan net472 805ns 0.346ns 1.34ns 0.0916 0 0 578 B
#5499 StartFinishScope net6.0 600ns 0.334ns 1.29ns 0.00976 0 0 696 B
#5499 StartFinishScope netcoreapp3.1 837ns 0.586ns 2.27ns 0.00927 0 0 696 B
#5499 StartFinishScope net472 962ns 0.319ns 1.19ns 0.104 0 0 658 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Slower :warning: Same allocations :heavy_check_mark:

Slower :warning: in #5499

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.TraceAnnotationsBenchmark.RunOnMethodBegin‑net6.0 1.276 643.74 821.54

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net6.0 644ns 0.238ns 0.857ns 0.00963 0 0 696 B
master RunOnMethodBegin netcoreapp3.1 981ns 1.75ns 6.54ns 0.0095 0 0 696 B
master RunOnMethodBegin net472 1.17μs 0.635ns 2.46ns 0.104 0 0 658 B
#5499 RunOnMethodBegin net6.0 821ns 0.285ns 1.1ns 0.00984 0 0 696 B
#5499 RunOnMethodBegin netcoreapp3.1 1.05μs 0.56ns 2.02ns 0.00912 0 0 696 B
#5499 RunOnMethodBegin net472 1.17μs 0.438ns 1.7ns 0.104 0 0 658 B

andrewlock avatar Apr 25 '24 13:04 andrewlock

Throughput/Crank Report:zap:

Throughput results for AspNetCoreSimpleController comparing the following branches/commits:

Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.

Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!

gantt
    title Throughput Linux x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5499) (11.986M)   : 0, 11986313
    master (11.916M)   : 0, 11916478
    benchmarks/2.9.0 (12.032M)   : 0, 12032037

    section Automatic
    This PR (5499) (7.901M)   : 0, 7900560
    master (8.109M)   : 0, 8109177
    benchmarks/2.9.0 (8.404M)   : 0, 8404279

    section Trace stats
    master (8.472M)   : 0, 8471982

    section Manual
    This PR (5499) (10.127M)   : 0, 10126645
    master (10.455M)   : 0, 10455228

    section Manual + Automatic
    This PR (5499) (7.609M)   : 0, 7609260
    master (7.651M)   : 0, 7650506

    section Version Conflict
    master (6.905M)   : 0, 6904520


gantt
    title Throughput Linux arm64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5499) (9.881M)   : 0, 9881022
    master (9.495M)   : 0, 9495268
    benchmarks/2.9.0 (9.620M)   : 0, 9619549

    section Automatic
    This PR (5499) (6.587M)   : 0, 6586647
    master (6.494M)   : 0, 6494221

    section Trace stats
    master (6.990M)   : 0, 6990390

    section Manual
    This PR (5499) (8.221M)   : 0, 8220777
    master (8.332M)   : 0, 8332179

    section Manual + Automatic
    This PR (5499) (6.123M)   : 0, 6122710
    master (6.244M)   : 0, 6244315

    section Version Conflict
    master (5.739M)   : 0, 5738657


gantt
    title Throughput Windows x64 (Total requests) 
    dateFormat  X
    axisFormat %s
    section Baseline
    This PR (5499) (9.841M)   : 0, 9841494
    master (9.906M)   : 0, 9905647
    benchmarks/2.9.0 (10.282M)   : 0, 10281996

    section Automatic
    This PR (5499) (7.124M)   : 0, 7123520
    master (7.188M)   : 0, 7187768
    benchmarks/2.9.0 (7.365M)   : 0, 7365461

    section Trace stats
    master (7.389M)   : 0, 7389061

    section Manual
    This PR (5499) (8.763M)   : 0, 8762747
    master (8.663M)   : 0, 8663172

    section Manual + Automatic
    This PR (5499) (6.866M)   : 0, 6865773
    master (6.905M)   : 0, 6904986

    section Version Conflict
    master (6.202M)   : 0, 6202215

andrewlock avatar Apr 25 '24 13:04 andrewlock

That's a first step. There are other things we might want to tackle/discussion, even go further:

  • if tracer jobs are not run on Profiler PRs. We should be able to update the PR checks status for those jobs. Otherwise, the PR will blocked due to required checks. We could remove the required tag, but I'm not sure that's a good idea. Completely noob question: could we dynamically set checks as required or not ? This way we could mark jobs as required or not depending on the changes. (another proposition, add one stage for profiler and one for tracer that would be required and checks specific jobs status. And depending on the PR change would just set as succeeded or as failed)
  • Maybe going a little bit farther: we could avoid running profiler benchmark on tracer/.. PRs and same for Profiler only changes.
  • If we do not run some tracer jobs, what about the trace_pipeline job? should we add the profiler jobs there too ?

gleocadie avatar Apr 26 '24 07:04 gleocadie

Closing as we already have in place several different strategies to do this

bouwkast avatar Oct 03 '25 22:10 bouwkast