runtime instrument type

[seankhliao]

My setup was a fairly simple one, expecting more or less the equivalent metrics from prometheus http.Handle("/prommetrics", promhttp.Handler()) for the go runtime.

	promExporter, _ := prometheus.InstallNewPipeline(prometheus.Config{})
	otelruntime.Start()
	http.Handle("/otelmetrics", promExporter)

The metrics recorded by runtime are of type metric.Int64UpDownSumObserver, which is additive, resulting in numbers that are out of place (eg 200k goroutines, 60GiB of allocated memory on a machine with 2GiB, ...) and graphs that look like

Whereas I was expecting the (in my opinion more useful):

The information read from go's stdlib provides a snapshot of the current go runtime state, and I was expecting this to be reflected in the metrics; instead, with the current behaviour, it will monotonically increase and to get the stats from a point in time requires the use of rate() or similar, losing resolution. The use of metric.Int64UpDownSumObserver indicates that the value is intended to go down at some point in time, but in current usage it never will

[jmacd]

We need to do something in the specification about this. Since Prometheus does not support UpDownSumObserver semantics, the Prometheus exporter should convert these to Gauge. There is a related OTEP here: https://github.com/open-telemetry/oteps/pull/118, but it does not state this specifically and it is mired in issues related to histogram vs gauge, not counter vs gauge. :grinning:

I will file a corresponding issue in the main repository.

[jmacd]

Thanks @seankhliao. It looks fairly straightforward to fix this support in the Prometheus exporter. https://github.com/open-telemetry/opentelemetry-go/issues/1210

[seankhliao]

thanks for looking into this