chainlink-common icon indicating copy to clipboard operation
chainlink-common copied to clipboard

INFOPLAT-1575 Implement Prometheus to OTel metrics Forwarder

Open pkcll opened this issue 11 months ago • 0 comments

Implement Prometheus to OTel metrics conversion and export.

Forwarder:

  • Gets metrics from prometheus.Gatherer and exports them to provided OTLP/GRPC endpoint. Uses piped Receiver and Exporter.

Receiver:

  • Wraps prometheusreceiver
  • Fetches prometheus metrics data via prometheus.Gatherer (same process memory, no HTTP calls)
  • Uses custom implementation of prometheus.scraper (from here https://github.com/pkcll/prometheus/pull/1) to shortcut HTTP request calls and fetch data from prometheus.Gatherer
  • Converts Prometheus metrics into OTel format using prometheusreceiver
  • Passes OTel metrics data to downstream OTel otlpexporter

Exporter:

  • Wraps otlpexporter
  • Receives metric data from the receiver
  • Export OTel metrics data to otel collector endpoint via otlpexporter

Upstream PR's

  • https://github.com/smartcontractkit/chainlink/pull/14597
    • https://github.com/smartcontractkit/chainlink/pull/16121

Downstream PR's

  • https://github.com/pkcll/opentelemetry-collector-contrib/pull/1
    • https://github.com/pkcll/prometheus/pull/3

Run Example

cd chainlink-common/pkg/promotel

# Start OTel Collector
docker run \
 -p 127.0.0.1:4317:4317 \
 otel/opentelemetry-collector-contrib:0.118.0

# Run example test
go test -timeout 30s -run ^TestExample$ github.com/smartcontractkit/chainlink-common/pkg/promotel/internal

OTel Collector logs with exported test metric TestExample_test_counter_metric

Metric #4
otelcollectoragent  | Descriptor:
otelcollectoragent  |      -> Name: TestExample_test_counter_metric
otelcollectoragent  |      -> Description:
otelcollectoragent  |      -> Unit:
otelcollectoragent  |      -> DataType: Sum
otelcollectoragent  |      -> IsMonotonic: true
otelcollectoragent  |      -> AggregationTemporality: Cumulative
otelcollectoragent  | NumberDataPoints #0
otelcollectoragent  | StartTimestamp: 2025-02-12 15:27:56.202 +0000 UTC
otelcollectoragent  | Timestamp: 2025-02-12 15:27:56.202 +0000 UTC
otelcollectoragent  | Value: 1.000000

pkcll avatar Jan 28 '25 15:01 pkcll