chainlink-common
chainlink-common copied to clipboard
INFOPLAT-1575 Implement Prometheus to OTel metrics Forwarder
Implement Prometheus to OTel metrics conversion and export.
Forwarder:
- Gets metrics from prometheus.Gatherer and exports them to provided OTLP/GRPC endpoint. Uses piped Receiver and Exporter.
Receiver:
- Wraps prometheusreceiver
- Fetches prometheus metrics data via
prometheus.Gatherer(same process memory, no HTTP calls) - Uses custom implementation of
prometheus.scraper(from here https://github.com/pkcll/prometheus/pull/1) to shortcut HTTP request calls and fetch data fromprometheus.Gatherer - Converts Prometheus metrics into OTel format using prometheusreceiver
- Passes OTel metrics data to downstream OTel otlpexporter
Exporter:
- Wraps otlpexporter
- Receives metric data from the receiver
- Export OTel metrics data to otel collector endpoint via otlpexporter
Upstream PR's
- https://github.com/smartcontractkit/chainlink/pull/14597
- https://github.com/smartcontractkit/chainlink/pull/16121
Downstream PR's
- https://github.com/pkcll/opentelemetry-collector-contrib/pull/1
- https://github.com/pkcll/prometheus/pull/3
Run Example
cd chainlink-common/pkg/promotel
# Start OTel Collector
docker run \
-p 127.0.0.1:4317:4317 \
otel/opentelemetry-collector-contrib:0.118.0
# Run example test
go test -timeout 30s -run ^TestExample$ github.com/smartcontractkit/chainlink-common/pkg/promotel/internal
OTel Collector logs with exported test metric TestExample_test_counter_metric
Metric #4
otelcollectoragent | Descriptor:
otelcollectoragent | -> Name: TestExample_test_counter_metric
otelcollectoragent | -> Description:
otelcollectoragent | -> Unit:
otelcollectoragent | -> DataType: Sum
otelcollectoragent | -> IsMonotonic: true
otelcollectoragent | -> AggregationTemporality: Cumulative
otelcollectoragent | NumberDataPoints #0
otelcollectoragent | StartTimestamp: 2025-02-12 15:27:56.202 +0000 UTC
otelcollectoragent | Timestamp: 2025-02-12 15:27:56.202 +0000 UTC
otelcollectoragent | Value: 1.000000