opentelemetry-specification Mark exemplars as stable.

Fixes #3756

Note: This is still draft for two reasons:

Would like to get the ExemplarReservoir configuration of views in some languages.
Would still like to override the specification so, by-default, if Traces are enabled and sampled, then Exemplars are enabled for otel metrics. See #3756 TODOs.

Changes

Marks Exemplar specification as stable.

For non-trivial changes, follow the change proposal process.

[X] Related issues #3756
[X] Related OTEP(s) #113
[ ] Links to the prototypes (when adding or changing features)
[ ] CHANGELOG.md file updated for non-trivial changes
[X] spec-compliance-matrix.md updated if necessary

Feb 08 '24 15:02 jsuereth

Are we in need of the usual 3 languages implemented before calling this stable? The spec matrix shows only Java, though .NET has most of the implementation (in preview). I can send an update to spec matrix to reflect .NET's status.

Feb 09 '24 00:02 cijothomas

Are we in need of the usual 3 languages implemented before calling this stable? The spec matrix shows only Java, though .NET has most of the implementation (in preview). I can send an update to spec matrix to reflect .NET's status.

Swift https://github.com/open-telemetry/opentelemetry-swift/tree/main/Sources/OpenTelemetrySdk/Metrics/Stable/Exemplar
PHP https://github.com/open-telemetry/opentelemetry-php/tree/main/src/SDK/Metrics/Exemplar
JavaScript https://github.com/open-telemetry/opentelemetry-js/blob/main/packages/sdk-metrics/src/exemplar/index.ts

Feb 14 '24 03:02 reyang

Are we in need of the usual 3 languages implemented before calling this stable? The spec matrix shows only Java, though .NET has most of the implementation (in preview). I can send an update to spec matrix to reflect .NET's status.

Swift https://github.com/open-telemetry/opentelemetry-swift/tree/main/Sources/OpenTelemetrySdk/Metrics/Stable/Exemplar

PHP https://github.com/open-telemetry/opentelemetry-php/tree/main/src/SDK/Metrics/Exemplar

JavaScript https://github.com/open-telemetry/opentelemetry-js/blob/main/packages/sdk-metrics/src/exemplar/index.ts

Those link does not tell much...For example, in JavaScript some exemplar related classes are there, but its not connected to SDK - its as good as saying JS does not support Exemplars... Which is why I was hoping every languages who implemented it will update the matrix, so we know if we have 3 languages implementing the spec. (or we can ask around as well :) )

OTel C++ also has some exemplars..

Feb 14 '24 17:02 cijothomas

Go has an experimental implementation. It does not expose any API for configuration outside of environment variables.

Feb 14 '24 17:02 MrAlias

This PR was marked stale due to lack of activity. It will be closed in 7 days.

Feb 22 '24 03:02 github-actions[bot]

Closed as inactive. Feel free to reopen if this PR is still being worked on.

Feb 29 '24 03:02 github-actions[bot]

This PR was marked stale due to lack of activity. It will be closed in 7 days.

Mar 08 '24 03:03 github-actions[bot]

The TC discussed this during the Mar. 13th, 2024 meeting:

There is only one blocking issue (more details can be found from #3756), @jack-berg is working on it #3943.
After PR #3943 is merged, we should be ready to move this PR forward. We're targeting mid Apr. for the Metrics Exemplar spec to be marked Stable (spec release will happen early May).

Mar 15 '24 19:03 reyang

This is now ready for review/merge

Mar 21 '24 14:03 jsuereth

What is the default behavior with respect to Exemplars?

There is this language "Exemplar sampling SHOULD be turned off by default." under https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk.md#exemplar. Does this mean, the default ExemplarFilter should be AlwaysOff? The https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/sdk.md#exemplarfilter section talks about the 3 possible values of ExemplarFilter, and marks TraceBased as default. The linked env variable section (https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/configuration/sdk-environment-variables.md#exemplar) also states TraceBased is the default.

This should be clarified (like editorial change only).

I also want to run some perf numbers from .NET, and share another related concern shortly.

Edit: Jack shared the same thing already!

Mar 21 '24 15:03 cijothomas

This PR was marked stale due to lack of activity. It will be closed in 7 days.

Mar 29 '24 03:03 github-actions[bot]

Closed as inactive. Feel free to reopen if this PR is still being worked on.

Apr 05 '24 03:04 github-actions[bot]

#3994 would hopefully be the last piece before we mark Exemplar spec stable. Please refer to the #3944 PR description for the Technical Committee discussion details.

@cijothomas would you check if it unblocks this PR? Thanks!

Apr 11 '24 02:04 reyang

#3994 would hopefully be the last piece before we mark Exemplar spec stable. Please refer to the #3944 PR description for the Technical Committee discussion details.

@cijothomas would you check if it unblocks this PR? Thanks!

Sure. Will check and post back here.

Apr 11 '24 17:04 cijothomas

@open-telemetry/specs-approvers @open-telemetry/specs-metrics-approvers @open-telemetry/technical-committee PTAL, I think this is ready to be merged by end of Fri. this week (Apr. 19th, 2024, PT).

@jack-berg @jmacd @jsuereth FYI

Apr 16 '24 15:04 reyang

https://github.com/open-telemetry/opentelemetry-dotnet/pull/5545#issue-2249411523 OTel .NET's performance hit when enabling exemplars is shown here. And OTel .NET is planning to make Exemplars opt-in for now.

@jack-berg Do you have some perf numbers from Java handy that can be shared?

Apr 19 '24 16:04 cijothomas

I measured the Lightstep metrics SDK implementation of exemplars here and the performance hit is consistent with what has been reported for .NET. For example, a fast-path that was 45ns becomes 60ns, for a 33% hit. I believe this is acceptable, and if there are applications for which a few 10s or 100s of nanoseconds of additional cost per metrics event are limiting, there may be other paths to optimization other than simply disabling exemplars.

https://github.com/lightstep/otel-launcher-go/pull/576#issuecomment-2067088223

Apr 19 '24 18:04 jmacd

Here are the benchmarks from the java repo. Sharing these without having recently dug into what elements of the exemplar implementation drive perf.

The benchmark tests a variety of dimension:

Single thread vs. multiple threads (in "Benchmark" column)
Configuration of attributes which receive measurements (in "Benchmark" column)
The instrument being recorded to (counter, histogram, etc, in the "(opBuilder)" column)
The sdk configuration (noop api, exemplars enabled, examplars disabled, temporality, in the "(sdk)" column)

Benchmark                                                                     (opBuilder)             (sdk)  Mode  Cnt     Score     Error   Units
MetricsBenchmarks.eightThreadsCommonLabelSet                               LongCounterAdd          API_ONLY  avgt   10     0.982 ±   0.034   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate                 LongCounterAdd          API_ONLY  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm            LongCounterAdd          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                      LongCounterAdd          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                               LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10  1241.016 ±  11.347   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate                 LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm            LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                      LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                               LongCounterAdd    SDK_CUMULATIVE  avgt   10  1235.555 ±  21.709   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate                 LongCounterAdd    SDK_CUMULATIVE  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm            LongCounterAdd    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                      LongCounterAdd    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                               LongCounterAdd         SDK_DELTA  avgt   10   983.321 ±  12.307   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate                 LongCounterAdd         SDK_DELTA  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm            LongCounterAdd         SDK_DELTA  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                      LongCounterAdd         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                             DoubleCounterAdd          API_ONLY  avgt   10     1.006 ±   0.072   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate               DoubleCounterAdd          API_ONLY  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm          DoubleCounterAdd          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                    DoubleCounterAdd          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                             DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10   961.077 ±   9.234   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate               DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm          DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                    DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                             DoubleCounterAdd    SDK_CUMULATIVE  avgt   10  1235.765 ±  16.153   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate               DoubleCounterAdd    SDK_CUMULATIVE  avgt   10     0.005 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm          DoubleCounterAdd    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                    DoubleCounterAdd    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                             DoubleCounterAdd         SDK_DELTA  avgt   10  1224.222 ±  27.974   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate               DoubleCounterAdd         SDK_DELTA  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm          DoubleCounterAdd         SDK_DELTA  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                    DoubleCounterAdd         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                        DoubleHistogramRecord          API_ONLY  avgt   10     2.716 ±   0.085   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate          DoubleHistogramRecord          API_ONLY  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm     DoubleHistogramRecord          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count               DoubleHistogramRecord          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                        DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10  1276.578 ±  10.122   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate          DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm     DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count               DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                        DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10  1270.384 ± 114.901   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate          DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10    14.956 ±  12.737  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm     DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10     2.439 ±   2.380    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count               DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10     1.000            counts
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.time                DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10     2.000                ms
MetricsBenchmarks.eightThreadsCommonLabelSet                        DoubleHistogramRecord         SDK_DELTA  avgt   10  1292.251 ±   4.334   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate          DoubleHistogramRecord         SDK_DELTA  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm     DoubleHistogramRecord         SDK_DELTA  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count               DoubleHistogramRecord         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                          LongHistogramRecord          API_ONLY  avgt   10     2.046 ±   0.006   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate            LongHistogramRecord          API_ONLY  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm       LongHistogramRecord          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                 LongHistogramRecord          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                          LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10  1313.262 ±   9.764   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate            LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm       LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                 LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                          LongHistogramRecord    SDK_CUMULATIVE  avgt   10  1290.082 ±  34.783   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate            LongHistogramRecord    SDK_CUMULATIVE  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm       LongHistogramRecord    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                 LongHistogramRecord    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsCommonLabelSet                          LongHistogramRecord         SDK_DELTA  avgt   10  1259.403 ±  39.749   ns/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate            LongHistogramRecord         SDK_DELTA  avgt   10    14.872 ±   2.581  MB/sec
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.alloc.rate.norm       LongHistogramRecord         SDK_DELTA  avgt   10     2.433 ±   0.366    B/op
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.count                 LongHistogramRecord         SDK_DELTA  avgt   10     1.000            counts
MetricsBenchmarks.eightThreadsCommonLabelSet:gc.time                  LongHistogramRecord         SDK_DELTA  avgt   10     2.000                ms
MetricsBenchmarks.eightThreadsSeparateLabelSets                            LongCounterAdd          API_ONLY  avgt   10     1.027 ±   0.020   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate              LongCounterAdd          API_ONLY  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm         LongCounterAdd          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count                   LongCounterAdd          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                            LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10   987.828 ±  23.329   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate              LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm         LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count                   LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                            LongCounterAdd    SDK_CUMULATIVE  avgt   10   969.901 ±  11.455   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate              LongCounterAdd    SDK_CUMULATIVE  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm         LongCounterAdd    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count                   LongCounterAdd    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                            LongCounterAdd         SDK_DELTA  avgt   10  1239.477 ±  23.283   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate              LongCounterAdd         SDK_DELTA  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm         LongCounterAdd         SDK_DELTA  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count                   LongCounterAdd         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                          DoubleCounterAdd          API_ONLY  avgt   10     0.964 ±   0.007   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate            DoubleCounterAdd          API_ONLY  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm       DoubleCounterAdd          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count                 DoubleCounterAdd          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                          DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10  1258.761 ±  21.210   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate            DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm       DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count                 DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                          DoubleCounterAdd    SDK_CUMULATIVE  avgt   10   972.530 ±  19.704   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate            DoubleCounterAdd    SDK_CUMULATIVE  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm       DoubleCounterAdd    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count                 DoubleCounterAdd    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                          DoubleCounterAdd         SDK_DELTA  avgt   10  1237.105 ±   7.030   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate            DoubleCounterAdd         SDK_DELTA  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm       DoubleCounterAdd         SDK_DELTA  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count                 DoubleCounterAdd         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                     DoubleHistogramRecord          API_ONLY  avgt   10     3.551 ±   0.129   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate       DoubleHistogramRecord          API_ONLY  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm  DoubleHistogramRecord          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count            DoubleHistogramRecord          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                     DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10  1413.131 ±  79.087   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate       DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm  DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count            DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                     DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10  1293.687 ±  32.239   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate       DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm  DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count            DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                     DoubleHistogramRecord         SDK_DELTA  avgt   10  1300.389 ±  21.223   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate       DoubleHistogramRecord         SDK_DELTA  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm  DoubleHistogramRecord         SDK_DELTA  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count            DoubleHistogramRecord         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                       LongHistogramRecord          API_ONLY  avgt   10     2.165 ±   0.196   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate         LongHistogramRecord          API_ONLY  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm    LongHistogramRecord          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count              LongHistogramRecord          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                       LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10  1321.396 ±  16.329   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate         LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm    LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count              LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets                       LongHistogramRecord    SDK_CUMULATIVE  avgt   10  1214.101 ±  50.256   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate         LongHistogramRecord    SDK_CUMULATIVE  avgt   10    12.611 ±   8.528  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm    LongHistogramRecord    SDK_CUMULATIVE  avgt   10     1.925 ±   1.395    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count              LongHistogramRecord    SDK_CUMULATIVE  avgt   10     1.000            counts
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.time               LongHistogramRecord    SDK_CUMULATIVE  avgt   10     2.000                ms
MetricsBenchmarks.eightThreadsSeparateLabelSets                       LongHistogramRecord         SDK_DELTA  avgt   10  1297.769 ±   5.424   ns/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate         LongHistogramRecord         SDK_DELTA  avgt   10     0.004 ±   0.001  MB/sec
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.alloc.rate.norm    LongHistogramRecord         SDK_DELTA  avgt   10     0.001 ±   0.001    B/op
MetricsBenchmarks.eightThreadsSeparateLabelSets:gc.count              LongHistogramRecord         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                                LongCounterAdd          API_ONLY  avgt   10     0.901 ±   0.008   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                                  LongCounterAdd          API_ONLY  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                             LongCounterAdd          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.oneThread:gc.count                                       LongCounterAdd          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                                LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10    17.664 ±   0.143   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                                  LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                             LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10    ≈ 10⁻⁵              B/op
MetricsBenchmarks.oneThread:gc.count                                       LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                                LongCounterAdd    SDK_CUMULATIVE  avgt   10    29.145 ±   0.822   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                                  LongCounterAdd    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                             LongCounterAdd    SDK_CUMULATIVE  avgt   10    ≈ 10⁻⁵              B/op
MetricsBenchmarks.oneThread:gc.count                                       LongCounterAdd    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                                LongCounterAdd         SDK_DELTA  avgt   10    28.836 ±   0.948   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                                  LongCounterAdd         SDK_DELTA  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                             LongCounterAdd         SDK_DELTA  avgt   10    ≈ 10⁻⁵              B/op
MetricsBenchmarks.oneThread:gc.count                                       LongCounterAdd         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                              DoubleCounterAdd          API_ONLY  avgt   10     0.897 ±   0.008   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                                DoubleCounterAdd          API_ONLY  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                           DoubleCounterAdd          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.oneThread:gc.count                                     DoubleCounterAdd          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                              DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10    18.270 ±   0.159   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                                DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                           DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10    ≈ 10⁻⁵              B/op
MetricsBenchmarks.oneThread:gc.count                                     DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                              DoubleCounterAdd    SDK_CUMULATIVE  avgt   10    29.956 ±   0.859   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                                DoubleCounterAdd    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                           DoubleCounterAdd    SDK_CUMULATIVE  avgt   10    ≈ 10⁻⁵              B/op
MetricsBenchmarks.oneThread:gc.count                                     DoubleCounterAdd    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                              DoubleCounterAdd         SDK_DELTA  avgt   10    29.145 ±   0.829   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                                DoubleCounterAdd         SDK_DELTA  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                           DoubleCounterAdd         SDK_DELTA  avgt   10    ≈ 10⁻⁵              B/op
MetricsBenchmarks.oneThread:gc.count                                     DoubleCounterAdd         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                         DoubleHistogramRecord          API_ONLY  avgt   10     2.532 ±   0.097   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                           DoubleHistogramRecord          API_ONLY  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                      DoubleHistogramRecord          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.oneThread:gc.count                                DoubleHistogramRecord          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                         DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10    33.472 ±   0.138   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                           DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                      DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10    ≈ 10⁻⁵              B/op
MetricsBenchmarks.oneThread:gc.count                                DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                         DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10   118.182 ±   0.482   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                           DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                      DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10    ≈ 10⁻⁴              B/op
MetricsBenchmarks.oneThread:gc.count                                DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                         DoubleHistogramRecord         SDK_DELTA  avgt   10   118.630 ±   0.833   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                           DoubleHistogramRecord         SDK_DELTA  avgt   10   192.914 ±   1.353  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                      DoubleHistogramRecord         SDK_DELTA  avgt   10    24.000 ±   0.001    B/op
MetricsBenchmarks.oneThread:gc.count                                DoubleHistogramRecord         SDK_DELTA  avgt   10     3.000            counts
MetricsBenchmarks.oneThread:gc.time                                 DoubleHistogramRecord         SDK_DELTA  avgt   10     5.000                ms
MetricsBenchmarks.oneThread                                           LongHistogramRecord          API_ONLY  avgt   10     1.913 ±   0.006   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                             LongHistogramRecord          API_ONLY  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                        LongHistogramRecord          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.oneThread:gc.count                                  LongHistogramRecord          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                           LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10    33.737 ±   0.439   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                             LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                        LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10    ≈ 10⁻⁵              B/op
MetricsBenchmarks.oneThread:gc.count                                  LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                           LongHistogramRecord    SDK_CUMULATIVE  avgt   10   117.504 ±   0.741   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                             LongHistogramRecord    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                        LongHistogramRecord    SDK_CUMULATIVE  avgt   10    ≈ 10⁻⁴              B/op
MetricsBenchmarks.oneThread:gc.count                                  LongHistogramRecord    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.oneThread                                           LongHistogramRecord         SDK_DELTA  avgt   10   118.460 ±   1.293   ns/op
MetricsBenchmarks.oneThread:gc.alloc.rate                             LongHistogramRecord         SDK_DELTA  avgt   10   193.199 ±   2.106  MB/sec
MetricsBenchmarks.oneThread:gc.alloc.rate.norm                        LongHistogramRecord         SDK_DELTA  avgt   10    24.000 ±   0.001    B/op
MetricsBenchmarks.oneThread:gc.count                                  LongHistogramRecord         SDK_DELTA  avgt   10     3.000            counts
MetricsBenchmarks.oneThread:gc.time                                   LongHistogramRecord         SDK_DELTA  avgt   10     5.000                ms
MetricsBenchmarks.recordToMultipleAttributes                               LongCounterAdd          API_ONLY  avgt   10     4.268 ±   0.029   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate                 LongCounterAdd          API_ONLY  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm            LongCounterAdd          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                      LongCounterAdd          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                               LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10    90.090 ±   1.211   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate                 LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10   338.722 ±   4.522  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm            LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10    32.000 ±   0.001    B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                      LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10     6.000            counts
MetricsBenchmarks.recordToMultipleAttributes:gc.time                       LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10     9.000                ms
MetricsBenchmarks.recordToMultipleAttributes                               LongCounterAdd    SDK_CUMULATIVE  avgt   10   141.212 ±   0.982   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate                 LongCounterAdd    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm            LongCounterAdd    SDK_CUMULATIVE  avgt   10    ≈ 10⁻⁴              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                      LongCounterAdd    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                               LongCounterAdd         SDK_DELTA  avgt   10   141.101 ±   0.590   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate                 LongCounterAdd         SDK_DELTA  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm            LongCounterAdd         SDK_DELTA  avgt   10    ≈ 10⁻⁴              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                      LongCounterAdd         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                             DoubleCounterAdd          API_ONLY  avgt   10     4.282 ±   0.042   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate               DoubleCounterAdd          API_ONLY  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm          DoubleCounterAdd          API_ONLY  avgt   10    ≈ 10⁻⁶              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                    DoubleCounterAdd          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                             DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10    92.942 ±   0.717   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate               DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10   328.316 ±   2.523  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm          DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10    32.000 ±   0.001    B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                    DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10     5.000            counts
MetricsBenchmarks.recordToMultipleAttributes:gc.time                     DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10     8.000                ms
MetricsBenchmarks.recordToMultipleAttributes                             DoubleCounterAdd    SDK_CUMULATIVE  avgt   10   142.815 ±   1.041   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate               DoubleCounterAdd    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm          DoubleCounterAdd    SDK_CUMULATIVE  avgt   10    ≈ 10⁻⁴              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                    DoubleCounterAdd    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                             DoubleCounterAdd         SDK_DELTA  avgt   10   142.041 ±   0.833   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate               DoubleCounterAdd         SDK_DELTA  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm          DoubleCounterAdd         SDK_DELTA  avgt   10    ≈ 10⁻⁴              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                    DoubleCounterAdd         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                        DoubleHistogramRecord          API_ONLY  avgt   10    12.945 ±   0.130   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate          DoubleHistogramRecord          API_ONLY  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm     DoubleHistogramRecord          API_ONLY  avgt   10    ≈ 10⁻⁵              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count               DoubleHistogramRecord          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                        DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10   177.686 ±   5.963   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate          DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm     DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10    ≈ 10⁻⁴              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count               DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                        DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10   612.016 ±  17.927   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate          DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm     DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10    ≈ 10⁻³              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count               DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                        DoubleHistogramRecord         SDK_DELTA  avgt   10   606.133 ±   2.685   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate          DoubleHistogramRecord         SDK_DELTA  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm     DoubleHistogramRecord         SDK_DELTA  avgt   10    ≈ 10⁻³              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count               DoubleHistogramRecord         SDK_DELTA  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                          LongHistogramRecord          API_ONLY  avgt   10    10.684 ±   0.073   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate            LongHistogramRecord          API_ONLY  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm       LongHistogramRecord          API_ONLY  avgt   10    ≈ 10⁻⁵              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                 LongHistogramRecord          API_ONLY  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                          LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10   176.199 ±   2.489   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate            LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10     0.001 ±   0.001  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm       LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10    ≈ 10⁻⁴              B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                 LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10       ≈ 0            counts
MetricsBenchmarks.recordToMultipleAttributes                          LongHistogramRecord    SDK_CUMULATIVE  avgt   10   613.344 ±  23.925   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate            LongHistogramRecord    SDK_CUMULATIVE  avgt   10   186.664 ±   6.977  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm       LongHistogramRecord    SDK_CUMULATIVE  avgt   10   120.000 ±   0.001    B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                 LongHistogramRecord    SDK_CUMULATIVE  avgt   10     3.000            counts
MetricsBenchmarks.recordToMultipleAttributes:gc.time                  LongHistogramRecord    SDK_CUMULATIVE  avgt   10     5.000                ms
MetricsBenchmarks.recordToMultipleAttributes                          LongHistogramRecord         SDK_DELTA  avgt   10   606.087 ±   3.686   ns/op
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate            LongHistogramRecord         SDK_DELTA  avgt   10   188.795 ±   1.145  MB/sec
MetricsBenchmarks.recordToMultipleAttributes:gc.alloc.rate.norm       LongHistogramRecord         SDK_DELTA  avgt   10   120.000 ±   0.001    B/op
MetricsBenchmarks.recordToMultipleAttributes:gc.count                 LongHistogramRecord         SDK_DELTA  avgt   10     3.000            counts
MetricsBenchmarks.recordToMultipleAttributes:gc.time                  LongHistogramRecord         SDK_DELTA  avgt   10     6.000                ms

There's a lot of noise here, so I've narrowed it down to show the time difference with a single thread and exemplars enabled and disabled:

Benchmark                                                                     (opBuilder)             (sdk)  Mode  Cnt     Score     Error   Units
MetricsBenchmarks.oneThread                                                LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10    17.664 ±   0.143   ns/op
MetricsBenchmarks.oneThread                                                LongCounterAdd    SDK_CUMULATIVE  avgt   10    29.145 ±   0.822   ns/op
MetricsBenchmarks.oneThread                                              DoubleCounterAdd  SDK_NO_EXEMPLARS  avgt   10    18.270 ±   0.159   ns/op
MetricsBenchmarks.oneThread                                              DoubleCounterAdd    SDK_CUMULATIVE  avgt   10    29.956 ±   0.859   ns/op
MetricsBenchmarks.oneThread                                         DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10    33.472 ±   0.138   ns/op
MetricsBenchmarks.oneThread                                         DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10   118.182 ±   0.482   ns/op
MetricsBenchmarks.oneThread                                           LongHistogramRecord  SDK_NO_EXEMPLARS  avgt   10    33.737 ±   0.439   ns/op
MetricsBenchmarks.oneThread                                           LongHistogramRecord    SDK_CUMULATIVE  avgt   10   117.504 ±   0.741   ns/op

There's no doubt a performance impact to enabling examplars, but to add some context:

The record path with exemplars disabled is very optimized. The impact of exemplars may look high as a percentage compared to the base, but isn't much in absolute terms.
Small improvements are probably possible for the histogram exemplar reservoir. We use linear search for the bucket index today, and should probably use binary search when buckets are greater than a particular size. Also, quickly glancing at the profile, the majority of the additional time spent is accessing the current nano time from the VM, which seems like it can be optimized.
Java has had exemplars enabled by default since we first published a stable metrics API / SDK years ago. We've had nobody complain about performance or the defaults.

Apr 19 '24 20:04 jack-berg

Benchmark after optimizing accessing the current nano time from the VM below. Much more in line with what we're seeing from .NET and Go.

Benchmark                                                 (opBuilder)             (sdk)  Mode  Cnt   Score    Error   Units
MetricsBenchmarks.oneThread                            LongCounterAdd  SDK_NO_EXEMPLARS  avgt   10  17.323 ±  0.309   ns/op
MetricsBenchmarks.oneThread                            LongCounterAdd    SDK_CUMULATIVE  avgt   10  28.126 ±  1.143   ns/op
MetricsBenchmarks.oneThread                     DoubleHistogramRecord  SDK_NO_EXEMPLARS  avgt   10  33.842 ±  1.248   ns/op
MetricsBenchmarks.oneThread                     DoubleHistogramRecord    SDK_CUMULATIVE  avgt   10  55.855 ±  2.418   ns/op

Apr 19 '24 20:04 jack-berg