dd-trace-dotnet
dd-trace-dotnet copied to clipboard
DataDog
Kevin/profiler in tests
Summary of changes
Reason for change
Implementation details
Test coverage
Other details
Datadog Report
Branch report: kevin/profiler_in_tests
Commit report: 94afb4b
Test service: dd-trace-dotnet
:x: 244 Failed (9 Known Flaky), 336272 Passed, 1531 Skipped, 41m 48.13s Wall Time :snowflake: 10 New Flaky
:x: Failed Tests (244)
This report shows up to 5 failed tests.
-
GraphQL2SchemaV0Tests.SubmitsTraces-Datadog.Trace.ClrProfiler.IntegrationTests- :snowflake: Known flaky - DetailsExpand for error
esults do not match. ifferences: eceived: GraphQL2Tests.SubmitsTraces.SchemaV0.received.txt erified: GraphQL2Tests.SubmitsTraces.SchemaV0.verified.txt eceived Content: { TraceId: Id_1, SpanId: Id_2, Name: aspnet_core.request, .. -
GraphQL2SchemaV0Tests.SubmitsTraces-Datadog.Trace.ClrProfiler.IntegrationTests- DetailsExpand for error
esults do not match. ifferences: eceived: GraphQL2Tests.SubmitsTraces.SchemaV0.received.txt erified: GraphQL2Tests.SubmitsTraces.SchemaV0.verified.txt eceived Content: { TraceId: Id_1, SpanId: Id_2, Name: aspnet_core.request, .. -
GraphQL3SchemaV0Tests.SubmitsTraces-Datadog.Trace.ClrProfiler.IntegrationTests- DetailsExpand for error
esults do not match. ifferences: eceived: GraphQL3Tests.SubmitsTraces.SchemaV0.received.txt erified: GraphQL3Tests.SubmitsTraces.SchemaV0.verified.txt eceived Content: { TraceId: Id_1, SpanId: Id_2, Name: aspnet_core.request, .. -
GraphQL3SchemaV0Tests.SubmitsTraces-Datadog.Trace.ClrProfiler.IntegrationTests- DetailsExpand for error
esults do not match. ifferences: eceived: GraphQL3Tests.SubmitsTraces.SchemaV0.received.txt erified: GraphQL3Tests.SubmitsTraces.SchemaV0.verified.txt eceived Content: { TraceId: Id_1, SpanId: Id_2, Name: aspnet_core.request, .. -
GraphQL4SchemaV0Tests.SubmitsTraces-Datadog.Trace.ClrProfiler.IntegrationTests- :snowflake: Known flaky - DetailsExpand for error
esults do not match. ifferences: eceived: GraphQL4Tests.SubmitsTraces.SchemaV0.received.txt erified: GraphQL4Tests.SubmitsTraces.SchemaV0.verified.txt eceived Content: { TraceId: Id_1, SpanId: Id_2, Name: aspnet_core.request, ..
New Flaky Tests (10)
-
SubmitsTraces-Datadog.Trace.ClrProfiler.IntegrationTests.AdoNet.NpgsqlCommandTests- Last FailureExpand for error
xpected exit code: 0, actual exit code: 139. -
SubmitsTraces-Datadog.Trace.ClrProfiler.IntegrationTests.AdoNet.NpgsqlCommandTests- Last FailureExpand for error
xpected exit code: 0, actual exit code: 139. -
SubmitTraces-Datadog.Trace.ClrProfiler.IntegrationTests.CI.XUnitEvpTests- Last FailureExpand for error
ssert.Equal() Failure xpected: 16 ctual: 0 -
SubmitsTraces-Datadog.Trace.ClrProfiler.IntegrationTests.GraphQL7SchemaV1Tests- Last FailureExpand for error
esults do not match. ifferences: eceived: GraphQL7Tests.SubmitsTraces.SchemaV1.received.txt erified: GraphQL7Tests.SubmitsTraces.SchemaV1.verified.txt eceived Content: { TraceId: Id_1, SpanId: Id_2, Name: aspnet_core.request, .. -
HttpClient_SubmitsTraces-Datadog.Trace.ClrProfiler.IntegrationTests.HttpMessageHandlerTests- Last FailureExpand for error
xpected exit code: 0, actual exit code: 134.
![datadog-ddstaging[bot] avatar](https://avatars.githubusercontent.com/u/365230?v=4 "Published by a pub.dev verified publisher"){kind=small}
Execution-Time Benchmarks Report :stopwatch:
Execution-time results for samples comparing the following branches/commits:
Execution-time benchmarks measure the whole time it takes to execute a program. And are intended to measure the one-off costs. Cases where the execution time results for the PR are worse than latest master results are shown in red. The following thresholds were used for comparing the execution times:
- Welch test with statistical test for significance of 5%
- Only results indicating a difference greater than 5% and 5 ms are considered.
Note that these results are based on a single point-in-time result for each branch. For full results, see the dashboard.
Graphs show the p99 interval based on the mean and StdDev of the test run, as well as the mean value of the run (shown as a diamond below the graph).
gantt
title Execution time (ms) FakeDbCommand (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5267) - mean (73ms) : 65, 82
. : milestone, 73,
master - mean (74ms) : 66, 83
. : milestone, 74,
section CallTarget+Inlining+NGEN
This PR (5267) - mean (988ms) : 966, 1010
. : milestone, 988,
master - mean (996ms) : 972, 1020
. : milestone, 996,
gantt
title Execution time (ms) FakeDbCommand (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5267) - mean (111ms) : 107, 115
. : milestone, 111,
master - mean (111ms) : 108, 115
. : milestone, 111,
section CallTarget+Inlining+NGEN
This PR (5267) - mean (714ms) : 690, 738
. : milestone, 714,
master - mean (717ms) : 686, 749
. : milestone, 717,
gantt
title Execution time (ms) FakeDbCommand (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5267) - mean (93ms) : 90, 97
. : milestone, 93,
master - mean (94ms) : 91, 97
. : milestone, 94,
section CallTarget+Inlining+NGEN
This PR (5267) - mean (664ms) : 643, 686
. : milestone, 664,
master - mean (676ms) : 641, 712
. : milestone, 676,
gantt
title Execution time (ms) HttpMessageHandler (.NET Framework 4.6.2)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5267) - mean (189ms) : 186, 192
. : milestone, 189,
master - mean (188ms) : 186, 190
. : milestone, 188,
section CallTarget+Inlining+NGEN
This PR (5267) - mean (1,062ms) : 1035, 1089
. : milestone, 1062,
master - mean (1,064ms) : 1037, 1091
. : milestone, 1064,
gantt
title Execution time (ms) HttpMessageHandler (.NET Core 3.1)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5267) - mean (271ms) : 266, 275
. : milestone, 271,
master - mean (271ms) : 268, 275
. : milestone, 271,
section CallTarget+Inlining+NGEN
This PR (5267) - mean (869ms) : 842, 897
. : milestone, 869,
master - mean (871ms) : 843, 899
. : milestone, 871,
gantt
title Execution time (ms) HttpMessageHandler (.NET 6)
dateFormat X
axisFormat %s
todayMarker off
section Baseline
This PR (5267) - mean (260ms) : 256, 263
. : milestone, 260,
master - mean (261ms) : 257, 265
. : milestone, 261,
section CallTarget+Inlining+NGEN
This PR (5267) - mean (857ms) : 835, 879
. : milestone, 857,
master - mean (857ms) : 833, 881
. : milestone, 857,
Throughput/Crank Report:zap:
Throughput results for AspNetCoreSimpleController comparing the following branches/commits:
Cases where throughput results for the PR are worse than latest master (5% drop or greater), results are shown in red.
Note that these results are based on a single point-in-time result for each branch. For full results, see one of the many, many dashboards!
gantt
title Throughput Linux x64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (5267) (10.815M) : 0, 10815128
master (11.007M) : 0, 11006674
benchmarks/2.9.0 (10.795M) : 0, 10795198
section Automatic
This PR (5267) (7.518M) : 0, 7518465
master (7.556M) : 0, 7555579
benchmarks/2.9.0 (8.048M) : 0, 8047571
section Trace stats
This PR (5267) (7.730M) : 0, 7729948
master (7.855M) : 0, 7855147
section Manual
This PR (5267) (9.514M) : 0, 9513668
master (9.556M) : 0, 9555551
section Manual + Automatic
This PR (5267) (7.151M) : 0, 7150532
master (7.141M) : 0, 7140945
section Version Conflict
This PR (5267) (6.385M) : 0, 6384701
master (6.452M) : 0, 6451971
gantt
title Throughput Linux arm64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (5267) (9.706M) : 0, 9705523
master (9.496M) : 0, 9495623
benchmarks/2.9.0 (9.570M) : 0, 9570288
section Automatic
This PR (5267) (6.612M) : 0, 6612242
master (6.550M) : 0, 6550088
section Trace stats
This PR (5267) (6.951M) : 0, 6951364
master (6.930M) : 0, 6929825
section Manual
This PR (5267) (8.280M) : 0, 8280192
master (8.294M) : 0, 8293659
section Manual + Automatic
This PR (5267) (6.244M) : 0, 6244380
master (6.274M) : 0, 6273873
section Version Conflict
This PR (5267) (5.645M) : 0, 5644703
master (5.748M) : 0, 5748183
gantt
title Throughput Windows x64 (Total requests)
dateFormat X
axisFormat %s
section Baseline
This PR (5267) (9.914M) : 0, 9914273
master (10.120M) : 0, 10120352
benchmarks/2.9.0 (10.670M) : 0, 10669848
section Automatic
This PR (5267) (7.044M) : 0, 7044118
master (7.343M) : 0, 7343410
benchmarks/2.9.0 (7.700M) : 0, 7699769
section Trace stats
This PR (5267) (7.463M) : 0, 7463422
master (7.630M) : 0, 7629966
section Manual
This PR (5267) (8.937M) : 0, 8937449
master (8.884M) : 0, 8883719
section Manual + Automatic
This PR (5267) (6.914M) : 0, 6914412
master (6.989M) : 0, 6989119
section Version Conflict
This PR (5267) (6.327M) : 0, 6326899
master (6.322M) : 0, 6322239
gantt
title Throughput Linux x64 (ASM) (Total requests)
dateFormat X
axisFormat %s
section Baseline
master (7.454M) : 0, 7454237
benchmarks/2.9.0 (7.800M) : 0, 7799916
section No attack
master (1.841M) : 0, 1841297
benchmarks/2.9.0 (3.202M) : 0, 3202059
section Attack
master (1.454M) : 0, 1454025
benchmarks/2.9.0 (2.532M) : 0, 2531516
section Blocking
master (3.195M) : 0, 3194705
section IAST default
master (6.463M) : 0, 6462928
section IAST full
master (5.595M) : 0, 5594918
section Base vuln
master (0.946M) : 0, 945501
section IAST vuln
master (0.849M) : 0, 849226