opentelemetry-dotnet icon indicating copy to clipboard operation
opentelemetry-dotnet copied to clipboard

Use lock for Histogram

Open utpilla opened this issue 2 years ago • 1 comments

Use lock for Histogram updates

Stress Test Results (high contention) with Prometheus Exporter scraping interval = 10s

Multiple threads updating the same Histogram MetricPoint in parallel

C:\opentelemetry-dotnet\test\OpenTelemetry.Tests.Stress.Metrics>dotnet run --framework net6.0 --configuration Release
Running (concurrency = 8, prometheusEndpoint = http://localhost:9184/metrics/), press <Esc> to stop...
2022-08-05T00:18:05.8285320Z Loops: 664,533,915, Loops/Second: 5,193,733, CPU Cycles/Loop: 3,124, RunwayTime (Seconds): 121
2022-08-05T00:18:06.0325689Z Loops: 665,604,237, Loops/Second: 5,170,911, CPU Cycles/Loop: 3,104, RunwayTime (Seconds): 121
2022-08-05T00:18:06.2391271Z Loops: 666,664,950, Loops/Second: 5,129,816, CPU Cycles/Loop: 3,126, RunwayTime (Seconds): 121
2022-08-05T00:18:06.4450077Z Loops: 667,725,432, Loops/Second: 5,185,127, CPU Cycles/Loop: 3,106, RunwayTime (Seconds): 121
2022-08-05T00:18:06.6519226Z Loops: 668,790,473, Loops/Second: 5,139,500, CPU Cycles/Loop: 3,074, RunwayTime (Seconds): 122
2022-08-05T00:18:06.8585657Z Loops: 669,851,220, Loops/Second: 5,144,074, CPU Cycles/Loop: 3,169, RunwayTime (Seconds): 122

Stress Test Results (low contention) with Prometheus Exporter scraping interval = 10s

Multiple threads updating random Histogram MetricPoints in parallel

C:\opentelemetry-dotnet\test\OpenTelemetry.Tests.Stress.Metrics>dotnet run --framework net6.0 --configuration Release
Running (concurrency = 8, prometheusEndpoint = http://localhost:9184/metrics/), press <Esc> to stop...
2022-08-04T23:02:23.9567130Z Loops: 2,429,503,031, Loops/Second: 20,119,653, CPU Cycles/Loop: 1,128, RunwayTime (Seconds): 120
2022-08-04T23:02:24.2277354Z Loops: 2,434,415,612, Loops/Second: 19,988,132, CPU Cycles/Loop: 1,130, RunwayTime (Seconds): 121
2022-08-04T23:02:24.4725452Z Loops: 2,439,784,407, Loops/Second: 19,949,797, CPU Cycles/Loop: 1,128, RunwayTime (Seconds): 121
2022-08-04T23:02:24.7065189Z Loops: 2,444,642,611, Loops/Second: 20,134,782, CPU Cycles/Loop: 1,129, RunwayTime (Seconds): 121
2022-08-04T23:02:24.9475497Z Loops: 2,449,492,448, Loops/Second: 19,901,301, CPU Cycles/Loop: 1,130, RunwayTime (Seconds): 121
2022-08-04T23:02:25.1765231Z Loops: 2,453,822,909, Loops/Second: 19,435,860, CPU Cycles/Loop: 1,119, RunwayTime (Seconds): 122

Benchmark Results

// * Summary *

BenchmarkDotNet=v0.13.1, OS=Windows 10.0.22000 Intel Core i7-9700 CPU 3.00GHz, 1 CPU, 8 logical and 8 physical cores .NET SDK=6.0.302 [Host] : .NET 6.0.7 (6.0.722.32202), X64 RyuJIT DefaultJob : .NET 6.0.7 (6.0.722.32202), X64 RyuJIT

Method BoundCount Mean Error StdDev
HistogramHotPath 10 46.44 ns 0.842 ns 0.746 ns
HistogramWith1LabelHotPath 10 97.67 ns 1.968 ns 3.447 ns
HistogramWith3LabelsHotPath 10 202.86 ns 4.035 ns 4.956 ns
HistogramWith5LabelsHotPath 10 292.78 ns 5.758 ns 7.282 ns
HistogramWith7LabelsHotPath 10 355.86 ns 7.033 ns 14.366 ns
HistogramHotPath 20 49.53 ns 1.006 ns 0.941 ns
HistogramWith1LabelHotPath 20 106.52 ns 2.099 ns 2.943 ns
HistogramWith3LabelsHotPath 20 207.65 ns 4.064 ns 5.139 ns
HistogramWith5LabelsHotPath 20 299.79 ns 5.967 ns 9.465 ns
HistogramWith7LabelsHotPath 20 358.21 ns 7.190 ns 14.023 ns
HistogramHotPath 50 59.17 ns 1.210 ns 2.055 ns
HistogramWith1LabelHotPath 50 115.68 ns 2.265 ns 3.023 ns
HistogramWith3LabelsHotPath 50 221.92 ns 4.318 ns 4.973 ns
HistogramWith5LabelsHotPath 50 310.69 ns 6.006 ns 11.571 ns
HistogramWith7LabelsHotPath 50 378.06 ns 7.383 ns 8.789 ns
HistogramHotPath 100 75.17 ns 1.533 ns 3.701 ns
HistogramWith1LabelHotPath 100 127.82 ns 2.528 ns 3.105 ns
HistogramWith3LabelsHotPath 100 235.38 ns 4.545 ns 10.258 ns
HistogramWith5LabelsHotPath 100 328.82 ns 6.550 ns 13.817 ns
HistogramWith7LabelsHotPath 100 393.21 ns 7.738 ns 10.592 ns

utpilla avatar Aug 04 '22 23:08 utpilla

Codecov Report

Merging #3547 (f44920c) into main (666a0c1) will increase coverage by 0.13%. The diff coverage is 100.00%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main    #3547      +/-   ##
==========================================
+ Coverage   87.04%   87.17%   +0.13%     
==========================================
  Files         275      275              
  Lines        9959     9951       -8     
==========================================
+ Hits         8669     8675       +6     
+ Misses       1290     1276      -14     
Impacted Files Coverage Δ
src/OpenTelemetry/Metrics/HistogramBuckets.cs 100.00% <ø> (ø)
src/OpenTelemetry/Metrics/MetricPoint.cs 85.60% <100.00%> (+0.60%) :arrow_up:
src/OpenTelemetry/ProviderExtensions.cs 81.81% <0.00%> (-9.10%) :arrow_down:
...Telemetry/Metrics/PeriodicExportingMetricReader.cs 72.72% <0.00%> (-5.46%) :arrow_down:
...tpListener/Internal/PrometheusCollectionManager.cs 80.48% <0.00%> (+2.43%) :arrow_up:
...metryProtocol/Implementation/ActivityExtensions.cs 94.50% <0.00%> (+3.29%) :arrow_up:
...xporter.OpenTelemetryProtocol/OtlpTraceExporter.cs 77.27% <0.00%> (+18.18%) :arrow_up:
...entation/ExportClient/OtlpGrpcTraceExportClient.cs 78.57% <0.00%> (+28.57%) :arrow_up:

codecov[bot] avatar Aug 04 '22 23:08 codecov[bot]

This PR was marked stale due to lack of activity and will be closed in 7 days. Commenting or Pushing will instruct the bot to automatically remove the label. This bot runs once per day.

github-actions[bot] avatar Aug 13 '22 03:08 github-actions[bot]

Closed as inactive. Feel free to reopen if this PR is still being worked on.

github-actions[bot] avatar Aug 20 '22 03:08 github-actions[bot]