dd-trace-dotnet icon indicating copy to clipboard operation
dd-trace-dotnet copied to clipboard

[ASM] Querystring obfuscation on the serialization thread to the agent instead

Open anna-git opened this issue 3 years ago • 10 comments

Summary of changes

Pushing away on the agent writer serializers threads the resolution of http.url tag, which implies obfuscation and some overload

Reason for change

Offloads the obfuscation's performance hit on the request thread for frameworks < netcoreapp3.1 and net461 and in case the non backtracking option port of net 7.0 for > netcoreapp3.1 has to be rolled back later.

WITH lazy / WITH regex backtracking (without .NET 7 regex port and without the non backtracking option)
With uncropped query string 2000 characters Tests with rate limit 20 000 req/sec Net core 5.0


| application                   |                |
| ----------------------------- | -------------- |
| CPU Usage (%)                 | 99             |
| Cores usage (%)               | 396            |
| Working Set (MB)              | 9,524          |
| Private Memory (MB)           | 9,810          |
| Build Time (ms)               | 4,080          |
| Start Time (ms)               | 556            |
| Published Size (KB)           | 93,751         |
| .NET Core SDK Version         | 5.0.408        |
| ASP.NET Core Version          | 5.0.17+02c6de4 |
| .NET Runtime Version          | 5.0.17+6a98414 |
| Max CPU Usage (%)             | 161            |
| Max Working Set (MB)          | 9,985          |
| Max GC Heap Size (MB)         | 9,112          |
| Max Number of Gen 0 GCs / sec | 2.00           |
| Max Number of Gen 1 GCs / sec | 2.00           |
| Max Number of Gen 2 GCs / sec | 1.00           |
| Max Time in GC (%)            | 21.00          |
| Max Gen 0 Size (B)            | 564,753,112    |
| Max Gen 1 Size (B)            | 1,093,353,848  |
| Max Gen 2 Size (B)            | 8,010,220,992  |
| Max LOH Size (B)              | 179,391,616    |
| Max POH Size (B)              | 1,358,848      |
| Max Allocation Rate (B/sec)   | 187,496,400    |
| Max GC Heap Fragmentation     | NaN            |
| # of Assemblies Loaded        | 141            |
| Max Exceptions (#/s)          | 1              |
| Max Lock Contention (#/s)     | 13             |
| Max ThreadPool Threads Count  | 10             |
| Max ThreadPool Queue Length   | 209            |
| Max ThreadPool Items (#/s)    | 22,568         |
| Max Active Timers             | 13             |
| IL Jitted (B)                 | 634,690        |
| Methods Jitted                | 7,356          |
| Max Requests per second       | 20,480         |
| Total Requests                | 5,036,319      |
| Max Active Requests           | 6              |
| Failed Requests               | 0              |


| load                |           |
| ------------------- | --------- |
| CPU Usage (%)       | 45        |
| Cores usage (%)     | 90        |
| Working Set (MB)    | 40        |
| Private Memory (MB) | 104       |
| Start Time (ms)     | 97        |
| First Request (ms)  | 193       |
| Requests            | 4,481,539 |
| Bad responses       | 0         |
| Mean latency (us)   | 7,167     |
| Max latency (us)    | 848,619   |
| Requests/sec        | 18,680    |
| Requests/sec (max)  | 59,009    |

WITHOUT lazy / WITH regex backtracking (without .NET 7 regex port and without the non backtracking option)
With uncropped query string 2000 characters Tests with rate limit 20 000 req/sec Net core 5.0


| application                   |                |
| ----------------------------- | -------------- |
| CPU Usage (%)                 | 100            |
| Cores usage (%)               | 401            |
| Working Set (MB)              | 246            |
| Private Memory (MB)           | 641            |
| Build Time (ms)               | 4,160          |
| Start Time (ms)               | 552            |
| Published Size (KB)           | 93,751         |
| .NET Core SDK Version         | 5.0.408        |
| ASP.NET Core Version          | 5.0.17+02c6de4 |
| .NET Runtime Version          | 5.0.17+6a98414 |
| Max CPU Usage (%)             | 181            |
| Max Working Set (MB)          | 257            |
| Max GC Heap Size (MB)         | 127            |
| Max Number of Gen 0 GCs / sec | 2.00           |
| Max Number of Gen 1 GCs / sec | 1.00           |
| Max Number of Gen 2 GCs / sec | 1.00           |
| Max Time in GC (%)            | 0.00           |
| Max Gen 0 Size (B)            | 23,761,600     |
| Max Gen 1 Size (B)            | 10,871,976     |
| Max Gen 2 Size (B)            | 11,849,536     |
| Max LOH Size (B)              | 25,905,800     |
| Max POH Size (B)              | 1,358,848      |
| Max Allocation Rate (B/sec)   | 52,254,360     |
| Max GC Heap Fragmentation     | NaN            |
| # of Assemblies Loaded        | 137            |
| Max Exceptions (#/s)          | 0              |
| Max Lock Contention (#/s)     | 2              |
| Max ThreadPool Threads Count  | 13             |
| Max ThreadPool Queue Length   | 236            |
| Max ThreadPool Items (#/s)    | 3,664          |
| Max Active Timers             | 7              |
| IL Jitted (B)                 | 620,687        |
| Methods Jitted                | 7,167          |
| Max Requests per second       | 2,288          |
| Total Requests                | 605,451        |
| Max Active Requests           | 13             |
| Failed Requests               | 0              |


| load                |         |
| ------------------- | ------- |
| CPU Usage (%)       | 7       |
| Cores usage (%)     | 14      |
| Working Set (MB)    | 39      |
| Private Memory (MB) | 104     |
| Start Time (ms)     | 89      |
| First Request (ms)  | 251     |
| Requests            | 539,297 |
| Bad responses       | 0       |
| Mean latency (us)   | 113,893 |
| Max latency (us)    | 338,939 |
| Requests/sec        | 2,245   |
| Requests/sec (max)  | 16,079  |

Implementation details

Add a new type, LazyOrString which allows to either work with a string or with a Lazy depending if obfuscation is needed

Test coverage

Other details

anna-git avatar Jul 28 '22 18:07 anna-git

Benchmarks Report :snail:

Benchmarks for #3020 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.116
  • 1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net472 716μs 536ns 2.08μs 0.357 0 0 3.18 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 456μs 182ns 703ns 0 0 0 2.58 KB
#3020 WriteAndFlushEnrichedTraces net472 723μs 300ns 1.16μs 0.361 0 0 3.18 KB
#3020 WriteAndFlushEnrichedTraces netcoreapp3.1 462μs 316ns 1.22μs 0 0 0 2.58 KB
Benchmarks.Trace.AppSecBodyBenchmark - Faster :tada: Same allocations :heavy_check_mark:

Faster :tada: in #3020

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.AppSecBodyBenchmark.BodyExtractorSimpleBody‑netcoreapp3.1 1.116 245.77 220.26

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net472 1.75μs 0.889ns 3.33ns 0.237 0 0 1.49 KB
master AllCycleSimpleBody netcoreapp3.1 1.87μs 1.37ns 4.94ns 0.0188 0 0 1.37 KB
master AllCycleMoreComplexBody net472 17μs 12.8ns 49.7ns 1.38 0.0252 0 8.75 KB
master AllCycleMoreComplexBody netcoreapp3.1 14.2μs 16.6ns 60ns 0.106 0 0 7.85 KB
master BodyExtractorSimpleBody net472 265ns 0.521ns 2.02ns 0.0573 0 0 361 B
master BodyExtractorSimpleBody netcoreapp3.1 246ns 0.311ns 1.16ns 0.00367 0 0 272 B
master BodyExtractorMoreComplexBody net472 14.5μs 16.1ns 62.5ns 1.21 0.0218 0 7.62 KB
master BodyExtractorMoreComplexBody netcoreapp3.1 11.9μs 11.8ns 44.1ns 0.0944 0 0 6.75 KB
#3020 AllCycleSimpleBody net472 1.73μs 1.92ns 7.42ns 0.236 0 0 1.49 KB
#3020 AllCycleSimpleBody netcoreapp3.1 1.76μs 1.73ns 6.69ns 0.0185 0 0 1.37 KB
#3020 AllCycleMoreComplexBody net472 17μs 11.4ns 41.2ns 1.39 0.0254 0 8.75 KB
#3020 AllCycleMoreComplexBody netcoreapp3.1 14.1μs 12.1ns 45.4ns 0.104 0 0 7.85 KB
#3020 BodyExtractorSimpleBody net472 250ns 0.164ns 0.615ns 0.0573 0 0 361 B
#3020 BodyExtractorSimpleBody netcoreapp3.1 220ns 0.215ns 0.805ns 0.0038 0 0 272 B
#3020 BodyExtractorMoreComplexBody net472 15.2μs 8.55ns 33.1ns 1.21 0.0153 0 7.62 KB
#3020 BodyExtractorMoreComplexBody netcoreapp3.1 11.9μs 9.92ns 37.1ns 0.0895 0 0 6.75 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: More allocations :warning:

More allocations :warning: in #3020

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.AspNetCoreBenchmark.SendRequest‑netcoreapp3.1 20.37 KB 20.49 KB 120 B 0.59%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
master SendRequest netcoreapp3.1 180μs 206ns 798ns 0.18 0 0 20.37 KB
#3020 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#3020 SendRequest netcoreapp3.1 176μs 72ns 259ns 0.264 0 0 20.49 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net472 1.61μs 0.4ns 1.55ns 0.126 0.000807 0 794 B
master ExecuteNonQuery netcoreapp3.1 1.23μs 0.769ns 2.88ns 0.011 0 0 824 B
#3020 ExecuteNonQuery net472 1.53μs 0.49ns 1.9ns 0.126 0.000772 0 794 B
#3020 ExecuteNonQuery netcoreapp3.1 1.28μs 0.302ns 1.13ns 0.0108 0 0 824 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net472 2.08μs 1.73ns 6.68ns 0.159 0 0 1 KB
master CallElasticsearch netcoreapp3.1 1.37μs 1.31ns 4.92ns 0.0137 0 0 984 B
master CallElasticsearchAsync net472 2.33μs 2.66ns 10.3ns 0.18 0 0 1.14 KB
master CallElasticsearchAsync netcoreapp3.1 1.43μs 0.949ns 3.68ns 0.0149 0 0 1.1 KB
#3020 CallElasticsearch net472 2.22μs 2.15ns 8.34ns 0.159 0 0 1 KB
#3020 CallElasticsearch netcoreapp3.1 1.36μs 1.88ns 7.28ns 0.0136 0 0 984 B
#3020 CallElasticsearchAsync net472 2.31μs 4.38ns 17ns 0.181 0 0 1.14 KB
#3020 CallElasticsearchAsync netcoreapp3.1 1.43μs 0.857ns 3.09ns 0.0151 0 0 1.1 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net472 2.37μs 2.46ns 9.54ns 0.2 0 0 1.26 KB
master ExecuteAsync netcoreapp3.1 1.56μs 0.679ns 2.54ns 0.0164 0 0 1.22 KB
#3020 ExecuteAsync net472 2.51μs 2.39ns 8.94ns 0.199 0 0 1.26 KB
#3020 ExecuteAsync netcoreapp3.1 1.62μs 2.33ns 8.73ns 0.0164 0 0 1.22 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net472 5μs 2.97ns 11.5ns 0.394 0 0 2.48 KB
master SendAsync netcoreapp3.1 3.48μs 0.993ns 3.71ns 0.0331 0 0 2.4 KB
#3020 SendAsync net472 4.97μs 11.6ns 44.9ns 0.394 0 0 2.48 KB
#3020 SendAsync netcoreapp3.1 3.49μs 7.66ns 29.6ns 0.0314 0 0 2.4 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 2.89μs 0.608ns 2.36ns 0.263 0 0 1.66 KB
master EnrichedLog netcoreapp3.1 2.25μs 1.09ns 4.22ns 0.0235 0 0 1.73 KB
#3020 EnrichedLog net472 2.87μs 1.8ns 6.96ns 0.263 0 0 1.66 KB
#3020 EnrichedLog netcoreapp3.1 2.37μs 0.833ns 3.12ns 0.0237 0 0 1.73 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 147μs 102ns 396ns 0.669 0.223 0 4.5 KB
master EnrichedLog netcoreapp3.1 112μs 120ns 451ns 0.0559 0 0 4.38 KB
#3020 EnrichedLog net472 146μs 169ns 655ns 0.654 0.218 0 4.5 KB
#3020 EnrichedLog netcoreapp3.1 110μs 188ns 729ns 0.0549 0 0 4.38 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.47μs 1.34ns 5.03ns 0.545 0.00274 0 3.43 KB
master EnrichedLog netcoreapp3.1 4.27μs 7.2ns 26.9ns 0.0517 0 0 3.8 KB
#3020 EnrichedLog net472 5.18μs 13.4ns 52ns 0.545 0.00265 0 3.43 KB
#3020 EnrichedLog netcoreapp3.1 4.19μs 8.06ns 31.2ns 0.0514 0 0 3.8 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net472 2.05μs 1.49ns 5.76ns 0.194 0 0 1.22 KB
master SendReceive netcoreapp3.1 1.69μs 1.64ns 6.35ns 0.016 0 0 1.21 KB
#3020 SendReceive net472 1.98μs 1.14ns 4.43ns 0.194 0 0 1.22 KB
#3020 SendReceive netcoreapp3.1 1.64μs 0.491ns 1.84ns 0.0163 0 0 1.21 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 4.74μs 4.36ns 16.9ns 0.328 0 0 2.08 KB
master EnrichedLog netcoreapp3.1 4.22μs 4.44ns 17.2ns 0.0232 0 0 1.69 KB
#3020 EnrichedLog net472 4.71μs 5.3ns 19.1ns 0.329 0 0 2.08 KB
#3020 EnrichedLog netcoreapp3.1 4.22μs 5.15ns 19.3ns 0.0212 0 0 1.69 KB
Benchmarks.Trace.SpanBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net472 785ns 0.0709ns 0.246ns 0.104 0 0 658 B
master StartFinishSpan netcoreapp3.1 753ns 0.211ns 0.789ns 0.00896 0 0 648 B
master StartFinishScope net472 1.06μs 0.116ns 0.402ns 0.117 0 0 738 B
master StartFinishScope netcoreapp3.1 903ns 0.367ns 1.42ns 0.0104 0 0 768 B
#3020 StartFinishSpan net472 835ns 0.358ns 1.39ns 0.105 0 0 658 B
#3020 StartFinishSpan netcoreapp3.1 726ns 0.412ns 1.6ns 0.00871 0 0 648 B
#3020 StartFinishScope net472 1.07μs 0.417ns 1.61ns 0.117 0 0 738 B
#3020 StartFinishScope netcoreapp3.1 963ns 0.241ns 0.934ns 0.0101 0 0 768 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net472 1.27μs 0.278ns 1.07ns 0.117 0 0 738 B
master RunOnMethodBegin netcoreapp3.1 1.04μs 0.293ns 1.06ns 0.0104 0 0 768 B
#3020 RunOnMethodBegin net472 1.2μs 0.706ns 2.74ns 0.117 0 0 738 B
#3020 RunOnMethodBegin netcoreapp3.1 991ns 0.258ns 0.998ns 0.0105 0 0 768 B

andrewlock avatar Jul 28 '22 19:07 andrewlock

Benchmarks Report :snail:

Benchmarks for #3020 compared to master:

  • All benchmarks have the same speed
  • 1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net472 713μs 316ns 1.18μs 0.355 0 0 3.18 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 467μs 404ns 1.56μs 0 0 0 2.58 KB
#3020 WriteAndFlushEnrichedTraces net472 712μs 393ns 1.36μs 0.355 0 0 3.18 KB
#3020 WriteAndFlushEnrichedTraces netcoreapp3.1 472μs 219ns 821ns 0 0 0 2.59 KB
Benchmarks.Trace.AppSecBodyBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net472 1.73μs 1.45ns 5.61ns 0.237 0 0 1.49 KB
master AllCycleSimpleBody netcoreapp3.1 1.79μs 2.84ns 11ns 0.0186 0 0 1.37 KB
master AllCycleMoreComplexBody net472 17μs 12.4ns 46.3ns 1.39 0.0255 0 8.75 KB
master AllCycleMoreComplexBody netcoreapp3.1 14μs 13.1ns 49ns 0.105 0 0 7.85 KB
master BodyExtractorSimpleBody net472 272ns 0.215ns 0.832ns 0.0573 0 0 361 B
master BodyExtractorSimpleBody netcoreapp3.1 221ns 0.326ns 1.26ns 0.00367 0 0 272 B
master BodyExtractorMoreComplexBody net472 15.1μs 13.2ns 47.5ns 1.21 0.015 0 7.62 KB
master BodyExtractorMoreComplexBody netcoreapp3.1 11.9μs 10ns 38.8ns 0.089 0 0 6.75 KB
#3020 AllCycleSimpleBody net472 1.65μs 2.69ns 10.1ns 0.237 0 0 1.49 KB
#3020 AllCycleSimpleBody netcoreapp3.1 1.76μs 2.84ns 11ns 0.0185 0 0 1.37 KB
#3020 AllCycleMoreComplexBody net472 16.8μs 25.8ns 100ns 1.38 0.0252 0 8.75 KB
#3020 AllCycleMoreComplexBody netcoreapp3.1 14.5μs 13.7ns 51.3ns 0.101 0 0 7.85 KB
#3020 BodyExtractorSimpleBody net472 256ns 0.195ns 0.704ns 0.0573 0 0 361 B
#3020 BodyExtractorSimpleBody netcoreapp3.1 224ns 0.351ns 1.31ns 0.00369 0 0 272 B
#3020 BodyExtractorMoreComplexBody net472 14.8μs 10.4ns 39ns 1.2 0.015 0 7.62 KB
#3020 BodyExtractorMoreComplexBody netcoreapp3.1 11.9μs 10.4ns 38.9ns 0.0893 0 0 6.75 KB
Benchmarks.Trace.AspNetCoreBenchmark - Unknown :shrug: More allocations :warning:

More allocations :warning: in #3020

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.AspNetCoreBenchmark.SendRequest‑netcoreapp3.1 20.37 KB 20.47 KB 104 B 0.51%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
master SendRequest netcoreapp3.1 178μs 126ns 490ns 0.178 0 0 20.37 KB
#3020 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#3020 SendRequest netcoreapp3.1 177μs 114ns 441ns 0.266 0 0 20.47 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net472 1.59μs 0.416ns 1.61ns 0.126 0.000797 0 794 B
master ExecuteNonQuery netcoreapp3.1 1.27μs 0.58ns 2.25ns 0.0113 0 0 824 B
#3020 ExecuteNonQuery net472 1.51μs 0.361ns 1.35ns 0.126 0.000761 0 794 B
#3020 ExecuteNonQuery netcoreapp3.1 1.29μs 0.541ns 2.1ns 0.0109 0 0 824 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net472 2.26μs 2.72ns 10.5ns 0.158 0 0 1 KB
master CallElasticsearch netcoreapp3.1 1.42μs 1.97ns 7.65ns 0.0129 0 0 984 B
master CallElasticsearchAsync net472 2.34μs 3.87ns 15ns 0.18 0 0 1.14 KB
master CallElasticsearchAsync netcoreapp3.1 1.46μs 1.74ns 6.74ns 0.0146 0 0 1.1 KB
#3020 CallElasticsearch net472 2.22μs 3.14ns 12.2ns 0.159 0 0 1 KB
#3020 CallElasticsearch netcoreapp3.1 1.46μs 1.63ns 6.11ns 0.0133 0 0 984 B
#3020 CallElasticsearchAsync net472 2.19μs 1.28ns 4.79ns 0.181 0 0 1.14 KB
#3020 CallElasticsearchAsync netcoreapp3.1 1.45μs 1.51ns 5.86ns 0.0153 0 0 1.1 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net472 2.45μs 2ns 7.73ns 0.2 0 0 1.26 KB
master ExecuteAsync netcoreapp3.1 1.58μs 1.12ns 4.32ns 0.0165 0 0 1.22 KB
#3020 ExecuteAsync net472 2.43μs 3.33ns 12.9ns 0.2 0 0 1.26 KB
#3020 ExecuteAsync netcoreapp3.1 1.57μs 1.48ns 5.75ns 0.0165 0 0 1.22 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net472 5.02μs 5.73ns 21.4ns 0.392 0 0 2.48 KB
master SendAsync netcoreapp3.1 3.33μs 2.35ns 8.79ns 0.0318 0 0 2.4 KB
#3020 SendAsync net472 4.98μs 6.35ns 23.8ns 0.394 0 0 2.48 KB
#3020 SendAsync netcoreapp3.1 3.35μs 2.8ns 10.5ns 0.0334 0 0 2.4 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 2.81μs 2.09ns 8.09ns 0.263 0 0 1.66 KB
master EnrichedLog netcoreapp3.1 2.35μs 0.818ns 3.06ns 0.0236 0 0 1.73 KB
#3020 EnrichedLog net472 2.93μs 1.36ns 5.28ns 0.264 0 0 1.66 KB
#3020 EnrichedLog netcoreapp3.1 2.24μs 1.04ns 4.03ns 0.0227 0 0 1.73 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 147μs 149ns 578ns 0.659 0.22 0 4.5 KB
master EnrichedLog netcoreapp3.1 112μs 171ns 662ns 0 0 0 4.38 KB
#3020 EnrichedLog net472 146μs 298ns 1.15μs 0.665 0.222 0 4.5 KB
#3020 EnrichedLog netcoreapp3.1 110μs 123ns 477ns 0.0549 0 0 4.38 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.43μs 5.38ns 20.1ns 0.545 0.00272 0 3.43 KB
master EnrichedLog netcoreapp3.1 4.3μs 2.24ns 8.69ns 0.0518 0 0 3.8 KB
#3020 EnrichedLog net472 5.32μs 6.95ns 26ns 0.546 0.00265 0 3.43 KB
#3020 EnrichedLog netcoreapp3.1 4.29μs 2.65ns 9.55ns 0.051 0 0 3.8 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net472 1.98μs 1.17ns 4.54ns 0.193 0 0 1.22 KB
master SendReceive netcoreapp3.1 1.7μs 0.708ns 2.65ns 0.0161 0 0 1.21 KB
#3020 SendReceive net472 2.01μs 0.964ns 3.73ns 0.193 0 0 1.22 KB
#3020 SendReceive netcoreapp3.1 1.65μs 0.486ns 1.82ns 0.0167 0 0 1.21 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 4.76μs 3.43ns 13.3ns 0.329 0 0 2.08 KB
master EnrichedLog netcoreapp3.1 4.1μs 4.54ns 17.6ns 0.0228 0 0 1.69 KB
#3020 EnrichedLog net472 4.69μs 6.54ns 25.3ns 0.33 0 0 2.08 KB
#3020 EnrichedLog netcoreapp3.1 4.28μs 4.96ns 19.2ns 0.0214 0 0 1.69 KB
Benchmarks.Trace.SpanBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net472 897ns 0.427ns 1.6ns 0.104 0 0 658 B
master StartFinishSpan netcoreapp3.1 770ns 0.323ns 1.21ns 0.00887 0 0 648 B
master StartFinishScope net472 1.04μs 0.44ns 1.65ns 0.117 0 0 738 B
master StartFinishScope netcoreapp3.1 883ns 0.228ns 0.822ns 0.0102 0 0 768 B
#3020 StartFinishSpan net472 829ns 0.344ns 1.29ns 0.104 0 0 658 B
#3020 StartFinishSpan netcoreapp3.1 738ns 0.195ns 0.701ns 0.0087 0 0 648 B
#3020 StartFinishScope net472 1.04μs 0.396ns 1.48ns 0.117 0 0 738 B
#3020 StartFinishScope netcoreapp3.1 904ns 0.359ns 1.34ns 0.0104 0 0 768 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net472 1.22μs 0.435ns 1.69ns 0.117 0 0 738 B
master RunOnMethodBegin netcoreapp3.1 979ns 0.316ns 1.23ns 0.0102 0 0 768 B
#3020 RunOnMethodBegin net472 1.16μs 0.468ns 1.81ns 0.117 0 0 738 B
#3020 RunOnMethodBegin netcoreapp3.1 993ns 0.17ns 0.611ns 0.0104 0 0 768 B

andrewlock avatar Jul 29 '22 09:07 andrewlock

Benchmarks Report :snail:

Benchmarks for #3020 compared to master:

  • 1 benchmarks are faster, with geometric mean 1.145
  • 1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net472 713μs 316ns 1.18μs 0.355 0 0 3.18 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 467μs 404ns 1.56μs 0 0 0 2.58 KB
#3020 WriteAndFlushEnrichedTraces net472 707μs 281ns 1.05μs 0.353 0 0 3.18 KB
#3020 WriteAndFlushEnrichedTraces netcoreapp3.1 462μs 172ns 667ns 0 0 0 2.58 KB
Benchmarks.Trace.AppSecBodyBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net472 1.73μs 1.45ns 5.61ns 0.237 0 0 1.49 KB
master AllCycleSimpleBody netcoreapp3.1 1.79μs 2.84ns 11ns 0.0186 0 0 1.37 KB
master AllCycleMoreComplexBody net472 17μs 12.4ns 46.3ns 1.39 0.0255 0 8.75 KB
master AllCycleMoreComplexBody netcoreapp3.1 14μs 13.1ns 49ns 0.105 0 0 7.85 KB
master BodyExtractorSimpleBody net472 272ns 0.215ns 0.832ns 0.0573 0 0 361 B
master BodyExtractorSimpleBody netcoreapp3.1 221ns 0.326ns 1.26ns 0.00367 0 0 272 B
master BodyExtractorMoreComplexBody net472 15.1μs 13.2ns 47.5ns 1.21 0.015 0 7.62 KB
master BodyExtractorMoreComplexBody netcoreapp3.1 11.9μs 10ns 38.8ns 0.089 0 0 6.75 KB
#3020 AllCycleSimpleBody net472 1.67μs 1.53ns 5.94ns 0.236 0 0 1.49 KB
#3020 AllCycleSimpleBody netcoreapp3.1 1.77μs 2.51ns 9.38ns 0.0184 0 0 1.37 KB
#3020 AllCycleMoreComplexBody net472 16.7μs 10ns 34.7ns 1.39 0.0252 0 8.75 KB
#3020 AllCycleMoreComplexBody netcoreapp3.1 14.1μs 17.7ns 68.6ns 0.106 0 0 7.85 KB
#3020 BodyExtractorSimpleBody net472 256ns 0.459ns 1.78ns 0.0574 0 0 361 B
#3020 BodyExtractorSimpleBody netcoreapp3.1 220ns 0.194ns 0.725ns 0.00366 0 0 272 B
#3020 BodyExtractorMoreComplexBody net472 14.7μs 12.1ns 47ns 1.21 0.0147 0 7.62 KB
#3020 BodyExtractorMoreComplexBody netcoreapp3.1 12.7μs 16.6ns 62.1ns 0.0886 0 0 6.75 KB
Benchmarks.Trace.AspNetCoreBenchmark - Unknown :shrug: More allocations :warning:

More allocations :warning: in #3020

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.AspNetCoreBenchmark.SendRequest‑netcoreapp3.1 20.37 KB 20.47 KB 104 B 0.51%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
master SendRequest netcoreapp3.1 178μs 126ns 490ns 0.178 0 0 20.37 KB
#3020 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#3020 SendRequest netcoreapp3.1 178μs 129ns 500ns 0.266 0 0 20.47 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net472 1.59μs 0.416ns 1.61ns 0.126 0.000797 0 794 B
master ExecuteNonQuery netcoreapp3.1 1.27μs 0.58ns 2.25ns 0.0113 0 0 824 B
#3020 ExecuteNonQuery net472 1.52μs 0.363ns 1.4ns 0.126 0.000759 0 794 B
#3020 ExecuteNonQuery netcoreapp3.1 1.29μs 1.31ns 5.08ns 0.011 0 0 824 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net472 2.26μs 2.72ns 10.5ns 0.158 0 0 1 KB
master CallElasticsearch netcoreapp3.1 1.42μs 1.97ns 7.65ns 0.0129 0 0 984 B
master CallElasticsearchAsync net472 2.34μs 3.87ns 15ns 0.18 0 0 1.14 KB
master CallElasticsearchAsync netcoreapp3.1 1.46μs 1.74ns 6.74ns 0.0146 0 0 1.1 KB
#3020 CallElasticsearch net472 2.15μs 3.29ns 12.7ns 0.159 0 0 1 KB
#3020 CallElasticsearch netcoreapp3.1 1.34μs 1.21ns 4.68ns 0.0134 0 0 984 B
#3020 CallElasticsearchAsync net472 2.36μs 6.61ns 24.7ns 0.181 0 0 1.14 KB
#3020 CallElasticsearchAsync netcoreapp3.1 1.48μs 2.38ns 9.22ns 0.0148 0 0 1.1 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net472 2.45μs 2ns 7.73ns 0.2 0 0 1.26 KB
master ExecuteAsync netcoreapp3.1 1.58μs 1.12ns 4.32ns 0.0165 0 0 1.22 KB
#3020 ExecuteAsync net472 2.43μs 3.82ns 14.8ns 0.199 0 0 1.26 KB
#3020 ExecuteAsync netcoreapp3.1 1.55μs 1.04ns 3.9ns 0.0163 0 0 1.22 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net472 5.02μs 5.73ns 21.4ns 0.392 0 0 2.48 KB
master SendAsync netcoreapp3.1 3.33μs 2.35ns 8.79ns 0.0318 0 0 2.4 KB
#3020 SendAsync net472 4.93μs 1.28ns 4.8ns 0.393 0 0 2.48 KB
#3020 SendAsync netcoreapp3.1 3.43μs 2.73ns 10.6ns 0.0323 0 0 2.4 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 2.81μs 2.09ns 8.09ns 0.263 0 0 1.66 KB
master EnrichedLog netcoreapp3.1 2.35μs 0.818ns 3.06ns 0.0236 0 0 1.73 KB
#3020 EnrichedLog net472 2.89μs 2.01ns 7.51ns 0.263 0 0 1.66 KB
#3020 EnrichedLog netcoreapp3.1 2.32μs 1.63ns 6.3ns 0.023 0 0 1.73 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 147μs 149ns 578ns 0.659 0.22 0 4.5 KB
master EnrichedLog netcoreapp3.1 112μs 171ns 662ns 0 0 0 4.38 KB
#3020 EnrichedLog net472 146μs 102ns 395ns 0.659 0.22 0 4.5 KB
#3020 EnrichedLog netcoreapp3.1 113μs 103ns 400ns 0.0565 0 0 4.38 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.43μs 5.38ns 20.1ns 0.545 0.00272 0 3.43 KB
master EnrichedLog netcoreapp3.1 4.3μs 2.24ns 8.69ns 0.0518 0 0 3.8 KB
#3020 EnrichedLog net472 5.43μs 2.01ns 7.78ns 0.544 0.00271 0 3.43 KB
#3020 EnrichedLog netcoreapp3.1 4.32μs 1.65ns 6.39ns 0.0522 0 0 3.8 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net472 1.98μs 1.17ns 4.54ns 0.193 0 0 1.22 KB
master SendReceive netcoreapp3.1 1.7μs 0.708ns 2.65ns 0.0161 0 0 1.21 KB
#3020 SendReceive net472 1.99μs 1.13ns 4.06ns 0.194 0 0 1.22 KB
#3020 SendReceive netcoreapp3.1 1.68μs 0.619ns 2.4ns 0.016 0 0 1.21 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 4.76μs 3.43ns 13.3ns 0.329 0 0 2.08 KB
master EnrichedLog netcoreapp3.1 4.1μs 4.54ns 17.6ns 0.0228 0 0 1.69 KB
#3020 EnrichedLog net472 4.66μs 5.07ns 19.6ns 0.33 0 0 2.08 KB
#3020 EnrichedLog netcoreapp3.1 4.22μs 3.72ns 14.4ns 0.0211 0 0 1.69 KB
Benchmarks.Trace.SpanBenchmark - Faster :tada: Same allocations :heavy_check_mark:

Faster :tada: in #3020

Benchmark base/diff Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472 1.145 896.85 783.40

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net472 897ns 0.427ns 1.6ns 0.104 0 0 658 B
master StartFinishSpan netcoreapp3.1 770ns 0.323ns 1.21ns 0.00887 0 0 648 B
master StartFinishScope net472 1.04μs 0.44ns 1.65ns 0.117 0 0 738 B
master StartFinishScope netcoreapp3.1 883ns 0.228ns 0.822ns 0.0102 0 0 768 B
#3020 StartFinishSpan net472 783ns 0.225ns 0.84ns 0.105 0 0 658 B
#3020 StartFinishSpan netcoreapp3.1 726ns 0.413ns 1.55ns 0.00875 0 0 648 B
#3020 StartFinishScope net472 1.01μs 0.356ns 1.38ns 0.117 0 0 738 B
#3020 StartFinishScope netcoreapp3.1 917ns 0.239ns 0.925ns 0.0106 0 0 768 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net472 1.22μs 0.435ns 1.69ns 0.117 0 0 738 B
master RunOnMethodBegin netcoreapp3.1 979ns 0.316ns 1.23ns 0.0102 0 0 768 B
#3020 RunOnMethodBegin net472 1.16μs 0.264ns 0.987ns 0.117 0 0 738 B
#3020 RunOnMethodBegin netcoreapp3.1 1.06μs 0.231ns 0.894ns 0.0102 0 0 768 B

andrewlock avatar Jul 29 '22 10:07 andrewlock

Benchmarks Report :snail:

Benchmarks for #3020 compared to master:

  • 1 benchmarks are slower, with geometric mean 1.157
  • 1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net472 715μs 276ns 1.07μs 0.355 0 0 3.18 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 482μs 337ns 1.3μs 0 0 0 2.58 KB
#3020 WriteAndFlushEnrichedTraces net472 725μs 332ns 1.24μs 0.361 0 0 3.18 KB
#3020 WriteAndFlushEnrichedTraces netcoreapp3.1 467μs 202ns 727ns 0 0 0 2.58 KB
Benchmarks.Trace.AppSecBodyBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net472 1.65μs 1.52ns 5.89ns 0.237 0 0 1.49 KB
master AllCycleSimpleBody netcoreapp3.1 1.84μs 1.45ns 5.82ns 0.0182 0 0 1.37 KB
master AllCycleMoreComplexBody net472 17.3μs 15.4ns 59.5ns 1.38 0.0248 0 8.75 KB
master AllCycleMoreComplexBody netcoreapp3.1 14.1μs 10.6ns 41.1ns 0.106 0 0 7.85 KB
master BodyExtractorSimpleBody net472 255ns 0.23ns 0.859ns 0.0574 0 0 361 B
master BodyExtractorSimpleBody netcoreapp3.1 222ns 0.259ns 1ns 0.0037 0 0 272 B
master BodyExtractorMoreComplexBody net472 16μs 93.7ns 828ns 1.21 0.0155 0 7.62 KB
master BodyExtractorMoreComplexBody netcoreapp3.1 11.7μs 18.8ns 67.9ns 0.0879 0 0 6.75 KB
#3020 AllCycleSimpleBody net472 1.75μs 1.6ns 5.98ns 0.237 0 0 1.49 KB
#3020 AllCycleSimpleBody netcoreapp3.1 1.79μs 1.78ns 6.67ns 0.0187 0 0 1.37 KB
#3020 AllCycleMoreComplexBody net472 16.7μs 9.88ns 38.3ns 1.39 0.0251 0 8.75 KB
#3020 AllCycleMoreComplexBody netcoreapp3.1 15.1μs 17ns 63.7ns 0.105 0 0 7.85 KB
#3020 BodyExtractorSimpleBody net472 257ns 0.301ns 1.13ns 0.0573 0 0 361 B
#3020 BodyExtractorSimpleBody netcoreapp3.1 233ns 0.384ns 1.44ns 0.00372 0 0 272 B
#3020 BodyExtractorMoreComplexBody net472 14.7μs 10.2ns 38.3ns 1.21 0.0221 0 7.62 KB
#3020 BodyExtractorMoreComplexBody netcoreapp3.1 11.9μs 13.6ns 49.2ns 0.0892 0 0 6.75 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: More allocations :warning:

More allocations :warning: in #3020

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.AspNetCoreBenchmark.SendRequest‑netcoreapp3.1 20.37 KB 20.47 KB 104 B 0.51%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
master SendRequest netcoreapp3.1 179μs 94.3ns 340ns 0.268 0 0 20.37 KB
#3020 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#3020 SendRequest netcoreapp3.1 178μs 191ns 740ns 0.266 0 0 20.47 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net472 1.57μs 0.492ns 1.91ns 0.126 0.000793 0 794 B
master ExecuteNonQuery netcoreapp3.1 1.27μs 0.461ns 1.6ns 0.011 0 0 824 B
#3020 ExecuteNonQuery net472 1.56μs 0.517ns 2ns 0.126 0.00079 0 794 B
#3020 ExecuteNonQuery netcoreapp3.1 1.27μs 0.549ns 1.9ns 0.0108 0 0 824 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net472 2.15μs 3.6ns 13.5ns 0.159 0 0 1 KB
master CallElasticsearch netcoreapp3.1 1.37μs 1.09ns 4.08ns 0.0132 0 0 984 B
master CallElasticsearchAsync net472 2.34μs 3.28ns 12.7ns 0.181 0 0 1.14 KB
master CallElasticsearchAsync netcoreapp3.1 1.48μs 1.79ns 6.72ns 0.0147 0 0 1.1 KB
#3020 CallElasticsearch net472 2.17μs 2.44ns 9.46ns 0.159 0 0 1 KB
#3020 CallElasticsearch netcoreapp3.1 1.41μs 1.08ns 4.03ns 0.0132 0 0 984 B
#3020 CallElasticsearchAsync net472 2.3μs 1.5ns 5.8ns 0.181 0 0 1.14 KB
#3020 CallElasticsearchAsync netcoreapp3.1 1.41μs 0.952ns 3.56ns 0.0149 0 0 1.1 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net472 2.44μs 8.84ns 34.2ns 0.2 0 0 1.26 KB
master ExecuteAsync netcoreapp3.1 1.57μs 1.59ns 6.15ns 0.0166 0 0 1.22 KB
#3020 ExecuteAsync net472 2.42μs 1.04ns 3.91ns 0.2 0 0 1.26 KB
#3020 ExecuteAsync netcoreapp3.1 1.68μs 0.654ns 2.36ns 0.0161 0 0 1.22 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net472 4.95μs 7.27ns 26.2ns 0.393 0 0 2.48 KB
master SendAsync netcoreapp3.1 3.37μs 2.7ns 10.5ns 0.032 0 0 2.4 KB
#3020 SendAsync net472 5.11μs 1.83ns 7.08ns 0.392 0 0 2.48 KB
#3020 SendAsync netcoreapp3.1 3.38μs 1.58ns 6.12ns 0.0336 0 0 2.4 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 2.8μs 0.836ns 3.13ns 0.264 0 0 1.66 KB
master EnrichedLog netcoreapp3.1 2.28μs 1.85ns 6.93ns 0.0228 0 0 1.73 KB
#3020 EnrichedLog net472 2.88μs 1.6ns 6.2ns 0.263 0 0 1.66 KB
#3020 EnrichedLog netcoreapp3.1 2.3μs 0.726ns 2.62ns 0.0229 0 0 1.73 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 146μs 113ns 437ns 0.665 0.222 0 4.5 KB
master EnrichedLog netcoreapp3.1 113μs 100ns 389ns 0.0566 0 0 4.38 KB
#3020 EnrichedLog net472 146μs 91.3ns 354ns 0.654 0.218 0 4.5 KB
#3020 EnrichedLog netcoreapp3.1 112μs 93.8ns 351ns 0 0 0 4.38 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.33μs 7.85ns 30.4ns 0.543 0.00265 0 3.43 KB
master EnrichedLog netcoreapp3.1 4.43μs 2.88ns 10.8ns 0.0516 0 0 3.8 KB
#3020 EnrichedLog net472 5.42μs 1.45ns 5.44ns 0.545 0.00273 0 3.43 KB
#3020 EnrichedLog netcoreapp3.1 4.32μs 1.68ns 6.29ns 0.0517 0 0 3.8 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net472 2.04μs 1.39ns 5.39ns 0.193 0 0 1.22 KB
master SendReceive netcoreapp3.1 1.61μs 2.35ns 8.47ns 0.0161 0 0 1.21 KB
#3020 SendReceive net472 2.03μs 0.638ns 2.39ns 0.193 0 0 1.22 KB
#3020 SendReceive netcoreapp3.1 1.67μs 0.648ns 2.43ns 0.016 0 0 1.21 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 4.66μs 5.28ns 19.8ns 0.33 0 0 2.08 KB
master EnrichedLog netcoreapp3.1 4.17μs 6.61ns 25.6ns 0.0228 0 0 1.69 KB
#3020 EnrichedLog net472 4.55μs 3.23ns 12.5ns 0.329 0 0 2.08 KB
#3020 EnrichedLog netcoreapp3.1 4.07μs 3.57ns 13.8ns 0.0223 0 0 1.69 KB
Benchmarks.Trace.SpanBenchmark - Slower :warning: Same allocations :heavy_check_mark:

Slower :warning: in #3020

Benchmark diff/base Base Median (ns) Diff Median (ns) Modality
Benchmarks.Trace.SpanBenchmark.StartFinishSpan‑net472 1.157 776.59 898.40

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net472 777ns 0.309ns 1.16ns 0.104 0 0 658 B
master StartFinishSpan netcoreapp3.1 782ns 0.295ns 1.1ns 0.00863 0 0 648 B
master StartFinishScope net472 1.09μs 0.461ns 1.78ns 0.117 0 0 738 B
master StartFinishScope netcoreapp3.1 949ns 0.371ns 1.34ns 0.0104 0 0 768 B
#3020 StartFinishSpan net472 898ns 0.147ns 0.569ns 0.105 0 0 658 B
#3020 StartFinishSpan netcoreapp3.1 736ns 0.229ns 0.858ns 0.00856 0 0 648 B
#3020 StartFinishScope net472 1.06μs 1.25ns 4.85ns 0.117 0 0 738 B
#3020 StartFinishScope netcoreapp3.1 865ns 0.348ns 1.35ns 0.01 0 0 768 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net472 1.23μs 6.49ns 35ns 0.117 0 0 738 B
master RunOnMethodBegin netcoreapp3.1 1.03μs 5.21ns 34.6ns 0.0106 0 0 768 B
#3020 RunOnMethodBegin net472 1.17μs 0.282ns 1.09ns 0.117 0 0 738 B
#3020 RunOnMethodBegin netcoreapp3.1 971ns 0.476ns 1.84ns 0.0103 0 0 768 B

andrewlock avatar Aug 01 '22 12:08 andrewlock

Benchmarks Report :snail:

Benchmarks for #3020 compared to master:

  • All benchmarks have the same speed
  • 1 benchmarks have more allocations

The following thresholds were used for comparing the benchmark speeds:

  • Mann–Whitney U test with statistical test for significance of 5%
  • Only results indicating a difference greater than 10% and 0.3 ns are considered.

Allocation changes below 0.5% are ignored.

Benchmark details

Benchmarks.Trace.AgentWriterBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master WriteAndFlushEnrichedTraces net472 715μs 276ns 1.07μs 0.355 0 0 3.18 KB
master WriteAndFlushEnrichedTraces netcoreapp3.1 482μs 337ns 1.3μs 0 0 0 2.58 KB
#3020 WriteAndFlushEnrichedTraces net472 728μs 603ns 2.34μs 0.363 0 0 3.18 KB
#3020 WriteAndFlushEnrichedTraces netcoreapp3.1 461μs 182ns 680ns 0 0 0 2.58 KB
Benchmarks.Trace.AppSecBodyBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master AllCycleSimpleBody net472 1.65μs 1.52ns 5.89ns 0.237 0 0 1.49 KB
master AllCycleSimpleBody netcoreapp3.1 1.84μs 1.45ns 5.82ns 0.0182 0 0 1.37 KB
master AllCycleMoreComplexBody net472 17.3μs 15.4ns 59.5ns 1.38 0.0248 0 8.75 KB
master AllCycleMoreComplexBody netcoreapp3.1 14.1μs 10.6ns 41.1ns 0.106 0 0 7.85 KB
master BodyExtractorSimpleBody net472 255ns 0.23ns 0.859ns 0.0574 0 0 361 B
master BodyExtractorSimpleBody netcoreapp3.1 222ns 0.259ns 1ns 0.0037 0 0 272 B
master BodyExtractorMoreComplexBody net472 16μs 93.7ns 828ns 1.21 0.0155 0 7.62 KB
master BodyExtractorMoreComplexBody netcoreapp3.1 11.7μs 18.8ns 67.9ns 0.0879 0 0 6.75 KB
#3020 AllCycleSimpleBody net472 1.73μs 1.49ns 5.77ns 0.237 0 0 1.49 KB
#3020 AllCycleSimpleBody netcoreapp3.1 1.83μs 2.43ns 9.43ns 0.0182 0 0 1.37 KB
#3020 AllCycleMoreComplexBody net472 16.7μs 12.1ns 46.8ns 1.38 0.0251 0 8.75 KB
#3020 AllCycleMoreComplexBody netcoreapp3.1 14μs 19.8ns 76.7ns 0.104 0 0 7.85 KB
#3020 BodyExtractorSimpleBody net472 259ns 0.084ns 0.314ns 0.0573 0 0 361 B
#3020 BodyExtractorSimpleBody netcoreapp3.1 227ns 0.268ns 1ns 0.00364 0 0 272 B
#3020 BodyExtractorMoreComplexBody net472 14.6μs 9.72ns 37.7ns 1.21 0.0218 0 7.62 KB
#3020 BodyExtractorMoreComplexBody netcoreapp3.1 12.2μs 15.4ns 57.5ns 0.0916 0 0 6.75 KB
Benchmarks.Trace.AspNetCoreBenchmark - Same speed :heavy_check_mark: More allocations :warning:

More allocations :warning: in #3020

Benchmark Base Allocated Diff Allocated Change Change %
Benchmarks.Trace.AspNetCoreBenchmark.SendRequest‑netcoreapp3.1 20.37 KB 20.47 KB 104 B 0.51%

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
master SendRequest netcoreapp3.1 179μs 94.3ns 340ns 0.268 0 0 20.37 KB
#3020 SendRequest net472 0ns 0ns 0ns 0 0 0 0 b
#3020 SendRequest netcoreapp3.1 179μs 54ns 209ns 0.267 0 0 20.47 KB
Benchmarks.Trace.DbCommandBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteNonQuery net472 1.57μs 0.492ns 1.91ns 0.126 0.000793 0 794 B
master ExecuteNonQuery netcoreapp3.1 1.27μs 0.461ns 1.6ns 0.011 0 0 824 B
#3020 ExecuteNonQuery net472 1.53μs 0.338ns 1.27ns 0.126 0.000765 0 794 B
#3020 ExecuteNonQuery netcoreapp3.1 1.28μs 0.424ns 1.58ns 0.0109 0 0 824 B
Benchmarks.Trace.ElasticsearchBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master CallElasticsearch net472 2.15μs 3.6ns 13.5ns 0.159 0 0 1 KB
master CallElasticsearch netcoreapp3.1 1.37μs 1.09ns 4.08ns 0.0132 0 0 984 B
master CallElasticsearchAsync net472 2.34μs 3.28ns 12.7ns 0.181 0 0 1.14 KB
master CallElasticsearchAsync netcoreapp3.1 1.48μs 1.79ns 6.72ns 0.0147 0 0 1.1 KB
#3020 CallElasticsearch net472 2.27μs 2.1ns 8.12ns 0.158 0 0 1 KB
#3020 CallElasticsearch netcoreapp3.1 1.42μs 1.39ns 5.38ns 0.0135 0 0 984 B
#3020 CallElasticsearchAsync net472 2.29μs 3.78ns 14.1ns 0.181 0 0 1.14 KB
#3020 CallElasticsearchAsync netcoreapp3.1 1.46μs 1.09ns 4.24ns 0.0147 0 0 1.1 KB
Benchmarks.Trace.GraphQLBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master ExecuteAsync net472 2.44μs 8.84ns 34.2ns 0.2 0 0 1.26 KB
master ExecuteAsync netcoreapp3.1 1.57μs 1.59ns 6.15ns 0.0166 0 0 1.22 KB
#3020 ExecuteAsync net472 2.49μs 0.548ns 2.05ns 0.2 0 0 1.26 KB
#3020 ExecuteAsync netcoreapp3.1 1.57μs 0.767ns 2.97ns 0.0166 0 0 1.22 KB
Benchmarks.Trace.HttpClientBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendAsync net472 4.95μs 7.27ns 26.2ns 0.393 0 0 2.48 KB
master SendAsync netcoreapp3.1 3.37μs 2.7ns 10.5ns 0.032 0 0 2.4 KB
#3020 SendAsync net472 5μs 0.991ns 3.71ns 0.393 0 0 2.48 KB
#3020 SendAsync netcoreapp3.1 3.36μs 1.06ns 3.81ns 0.0318 0 0 2.4 KB
Benchmarks.Trace.ILoggerBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 2.8μs 0.836ns 3.13ns 0.264 0 0 1.66 KB
master EnrichedLog netcoreapp3.1 2.28μs 1.85ns 6.93ns 0.0228 0 0 1.73 KB
#3020 EnrichedLog net472 2.88μs 1.88ns 7.29ns 0.264 0 0 1.66 KB
#3020 EnrichedLog netcoreapp3.1 2.33μs 0.954ns 3.57ns 0.0232 0 0 1.73 KB
Benchmarks.Trace.Log4netBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 146μs 113ns 437ns 0.665 0.222 0 4.5 KB
master EnrichedLog netcoreapp3.1 113μs 100ns 389ns 0.0566 0 0 4.38 KB
#3020 EnrichedLog net472 145μs 263ns 1.02μs 0.66 0.22 0 4.5 KB
#3020 EnrichedLog netcoreapp3.1 110μs 67ns 251ns 0.0544 0 0 4.38 KB
Benchmarks.Trace.NLogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 5.33μs 7.85ns 30.4ns 0.543 0.00265 0 3.43 KB
master EnrichedLog netcoreapp3.1 4.43μs 2.88ns 10.8ns 0.0516 0 0 3.8 KB
#3020 EnrichedLog net472 5.47μs 1.55ns 5.8ns 0.544 0.00272 0 3.43 KB
#3020 EnrichedLog netcoreapp3.1 4.26μs 2.01ns 7.26ns 0.0512 0 0 3.8 KB
Benchmarks.Trace.RedisBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master SendReceive net472 2.04μs 1.39ns 5.39ns 0.193 0 0 1.22 KB
master SendReceive netcoreapp3.1 1.61μs 2.35ns 8.47ns 0.0161 0 0 1.21 KB
#3020 SendReceive net472 2.02μs 1.07ns 4.14ns 0.194 0 0 1.22 KB
#3020 SendReceive netcoreapp3.1 1.63μs 1.51ns 5.84ns 0.0163 0 0 1.21 KB
Benchmarks.Trace.SerilogBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master EnrichedLog net472 4.66μs 5.28ns 19.8ns 0.33 0 0 2.08 KB
master EnrichedLog netcoreapp3.1 4.17μs 6.61ns 25.6ns 0.0228 0 0 1.69 KB
#3020 EnrichedLog net472 4.71μs 7.26ns 28.1ns 0.329 0 0 2.08 KB
#3020 EnrichedLog netcoreapp3.1 4.13μs 3.61ns 14ns 0.0225 0 0 1.69 KB
Benchmarks.Trace.SpanBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master StartFinishSpan net472 777ns 0.309ns 1.16ns 0.104 0 0 658 B
master StartFinishSpan netcoreapp3.1 782ns 0.295ns 1.1ns 0.00863 0 0 648 B
master StartFinishScope net472 1.09μs 0.461ns 1.78ns 0.117 0 0 738 B
master StartFinishScope netcoreapp3.1 949ns 0.371ns 1.34ns 0.0104 0 0 768 B
#3020 StartFinishSpan net472 858ns 0.123ns 0.478ns 0.104 0 0 658 B
#3020 StartFinishSpan netcoreapp3.1 729ns 0.187ns 0.723ns 0.00875 0 0 648 B
#3020 StartFinishScope net472 1.09μs 0.362ns 1.4ns 0.117 0 0 738 B
#3020 StartFinishScope netcoreapp3.1 858ns 0.433ns 1.62ns 0.0103 0 0 768 B
Benchmarks.Trace.TraceAnnotationsBenchmark - Same speed :heavy_check_mark: Same allocations :heavy_check_mark:

Raw results

Branch Method Toolchain Mean StdError StdDev Gen 0 Gen 1 Gen 2 Allocated
master RunOnMethodBegin net472 1.23μs 6.49ns 35ns 0.117 0 0 738 B
master RunOnMethodBegin netcoreapp3.1 1.03μs 5.21ns 34.6ns 0.0106 0 0 768 B
#3020 RunOnMethodBegin net472 1.27μs 0.589ns 2.28ns 0.117 0 0 738 B
#3020 RunOnMethodBegin netcoreapp3.1 976ns 0.281ns 1.05ns 0.0103 0 0 768 B

andrewlock avatar Aug 01 '22 16:08 andrewlock

Code Coverage Report :bar_chart:

:heavy_check_mark: Merging #3020 into master will not change line coverage :heavy_check_mark: Merging #3020 into master will not change branch coverage :heavy_check_mark: Merging #3020 into master will will decrease complexity by 20

master #3020 Change
Lines 16837 / 22853 16843 / 22854
Lines % 74% 74% 0% :heavy_check_mark:
Branches 10034 / 14174 10008 / 14148
Branches % 71% 71% 0% :heavy_check_mark:
Complexity 15236 15216 -20 :heavy_check_mark:

View the full report for further details:

Datadog.Trace Breakdown :heavy_check_mark:

master #3020 Change
Lines % 74% 74% 0% :heavy_check_mark:
Branches % 71% 71% 0% :heavy_check_mark:
Complexity 15236 15216 -20 :heavy_check_mark:

The following classes have significant coverage changes.

File Line coverage change Branch coverage change Complexity change
Datadog.Trace.Ci.Processors.OriginTagTraceProcessor -9% :no_entry: -17% :no_entry: 0 :heavy_check_mark:
Datadog.Trace.Util.Http.HttpRequestUtils 0% :heavy_check_mark: 8% :heavy_check_mark: 2 :no_entry:
Datadog.Trace.Ci.CIVisibility 7% :heavy_check_mark: 7% :heavy_check_mark: 0 :heavy_check_mark:

The following classes were added in #3020:

File Line coverage Branch coverage Complexity
Datadog.Trace.Tagging.LazyOrString 100% 100% 5

View the full reports for further details:

andrewlock avatar Aug 01 '22 16:08 andrewlock

  • Any risks of dropping spans in case of a lot of serialization work to do? Or any other impact; I haven't looked that part of the code to check, so real question?

I'm not sure I completely understand your question. Most of the other tracers implemented a mechanism like this. It has a double advantage:

  • work is off the critical path of the request,
  • if trace is dropped because of high volumes, then the work of obfuscation is not done

robertpi avatar Aug 05 '22 15:08 robertpi

  • if trace is dropped because of high volumes, then the work of obfuscation is not done

After investigating the trace serialization pipeline, this is unfortunately not true. Assuming that each of our double buffers is available for writing, we will first serialize the trace (including the tags) to see if there is room in the buffer, before determining whether a buffer is full.

  • work is off the critical path of the request,

In general I agree and think we should aim to achieve this. However, if we're not careful we may end up in the following scenario:

  • The current (1 of 2) buffer is not locked
  • We serialize the tags (possibly causing an obfuscation operation that times out at 100ms) and then learn that the resulting size is too large for the first buffer
  • We switch buffers
  • We serialize the tags again (possibly causing an obfuscation operation that times out at 100ms)
  • If there's room in the second buffer, we send the trace. If not, we drop the trace.

In the worst case, we cause the trace serialization thread to run 2x of the regex timeout and maybe we keep the trace. I think we can mitigate this by updating the interaction between AgentWriter.SerializeTrace and SpanBuffer.TryWrite to cache the results of the serialization so the worst case becomes 1x of the regex timeout.

zacharycmontoya avatar Aug 05 '22 19:08 zacharycmontoya

🤔 , also can this impact stats computation? If the regex is long to process, it blocks the serialization thread, then the buckets for the stats will be empty for the time of the serialization? I'll defer all that to @zacharycmontoya to correct everything I say.

It's true that in the current pipeline, a long-running trace serialization would also block our ability to put stats into the correct duration bucket. I think the best course of action is to do one of the following:

  • Create a stats point on the main thread before pushing the spans to the trace serialization thread
  • Immediately push the spans to a stats computation thread, where the stats points will be created, and once that is done push the spans onto the trace serialization thread

One reason why a separate thread may be preferred is that we need to do some additional normalization/obfuscation on span fields to create the correct stats buckets. The most expensive operations will be obfuscating resource names for SQL, Redis, other databases

zacharycmontoya avatar Aug 05 '22 19:08 zacharycmontoya

push the spans to a stats computation thread, where the stats points will be created, and once that is done push the spans onto the trace serialization thread

Sounds like we need to design a multi-threaded span processing pipeline!

obfuscate -> compute stats -> serialize -> send

lucaspimentel avatar Aug 12 '22 20:08 lucaspimentel