WIP Add runtime/metrics source option to instrumentation/runtime metrics

[jmacd]

Fixes #2624

The code in instrumentation/metrics was contributed a long time ago. The Go team has since introduced runtime metrics in an official way. This PR adds optional support for the new metrics. There is a problem in the old implementation because it was written before the current SDK specification; it is no longer correct to use synchronous instruments from async callbacks--where there are more than one reader concerned. This leaves the old code as-is and focuses on the desired future outcome without changing current default behavior.

I propose that we deprecate the old and adopt the new eventually, however there are a number of gotchas.

There are four histogram instruments, but we lack a way to record them asynchronously. Moreover, one is a Gauge Histogram which OTel has not specified.

https://github.com/open-telemetry/opentelemetry-specification/issues/2713 https://github.com/open-telemetry/opentelemetry-specification/issues/2714

There are questions about what is a UpDownCounter and what is a Gauge. They are all currently UpDownCounters, but that is no guarantee. Presently, if a non-cumulative metric is Uint64, it becomes UpDownCounter, and if the value is Float64 it becomes a Gauge. We have no way to test this Gauge path presently because the go-1.19 runtime currently produces all Uint64 metrics and Float64Hist metrics, no Float64 metrics.

Lastly, there are two cases in 1.19 where there are duplicate metric names, after removing units. This creates a technical problem for OTLP -> Prometheus -> OTLP; currently, the only way to do this that satisfies both groups' specifications is to use an empty unit string when the unit is in the metric name itself. Thus, when there is a conflict of this nature, the units are kept in the name and the OTel unit is empty. This is an area for development between OpenTelemetry and OpenMetrics/Prometheus.

[codecov[bot]]

Codecov Report

Merging #2643 (012c750) into main (bbbd638) will increase coverage by 0.0%. The diff coverage is 80.6%.

@@          Coverage Diff           @@
##            main   #2643    +/-   ##
======================================
  Coverage   74.4%   74.5%            
======================================
  Files        145     146     +1     
  Lines       6681    6828   +147     
======================================
+ Hits        4973    5087   +114     
- Misses      1561    1591    +30     
- Partials     147     150     +3     

Impacted Files	Coverage Δ
instrumentation/runtime/builtin.go	77.9% <77.9%> (ø)
instrumentation/runtime/runtime.go	75.1% <100.0%> (+1.1%)	:arrow_up:
samplers/jaegerremote/sampler_remote.go	85.4% <0.0%> (-2.0%)	:arrow_down:

[jmacd]

I feel this deserves more attention, and I don't feel the group has sufficient bandwidth while the metrics Alpha is still under development. I will bring this up for discussion, meanwhile will be testing a new package copied from this PR in the lightstep/otel-launcher-go repo.

[jmacd]

Lightstep will work to submit their downstream fork of this code to the upstream repository. For now, see https://github.com/lightstep/otel-launcher-go/tree/main/lightstep/instrumentation