New component: Prometheus Remote Write receiver

[tomatopunk]

The purpose and use-cases of the new component

The new component, Prometheus remote write receive, is designed to enable OpenTelemetry collectors to receive metrics data from Prometheus instances via the remote write protocol. This allows for seamless integration of Prometheus metrics into the OpenTelemetry ecosystem, enabling centralized metrics collection, processing, and exporting to various backends.

Example configuration for the component

receivers:
  prometheusremotewrite:
    endpoint: "0.0.0.0:12345"  # The address to listen on for Prometheus remote write 

Telemetry data types supported

metric

Is this a vendor-specific component?

• [ ] This is a vendor-specific component
• [ ] If this is a vendor-specific component, I am a member of the OpenTelemetry organization.
• [ ] If this is a vendor-specific component, I am proposing to contribute and support it as a representative of the vendor.

Code Owner(s)

@tomatopunk

Sponsor (optional)

@andrzej-stencel

Additional context

The tracking issue #10358, #14751 was closed because this component did not have a sponsor.

[dashpole]

See https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/14751#issuecomment-1348879991. The lack of transactionality in PRW 1.0 makes it difficult to implement a working receiver for PRW. IMO, we should wait to introduce this until PRW 2.0, which was just released as experimental: https://prometheus.io/docs/specs/remote_write_spec_2_0/. It solves the issues raised in the previous proposal.

[ChrsMark]

Example configuration for the component

receivers:
prometheusremotewrite:
   endpoint: "0.0.0.0:12345"  # The address to listen on for Prometheus remote write 

Should we consider tls+https support as well instead of just plain http?

[andrzej-stencel]

Thanks @dashpole, this is very valuable information.

Let's make sure the new receiver supports the PRW 2.0 spec.

[vipinvkmenon]

Looking forward to this feature. Id be more than happy to test and provide feedbacks if any once merged

[JustinMason]

https://github.com/open-telemetry/opentelemetry-collector-contrib/pull/34196

I have a great initial implementation of a remoteWrite receiver, this foundation implemented all the configurations supported by the oltp receiver. So HTTP, gRPC, CORS, TLS and mTLS, and auth settings are all in there.

This mirrored the https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver/otlpreceiver and its initial implementation is limited to sum and gauge metric mapping, but it is stable. The receiver has been running reliability in a production environment getting remoteWrites from prometheus and Victoria metrics.

[rakesh-p]

until this remotewrite receiver is ready, is there no other way to send metrics from prometheus to opentelemetry collector please?

we have several k8s clusters and each with its own prometheus running to collect the metrics from the cluster. we would like to send the metrics collected by all prometheus instances (1 in each cluster) to central opentelemetry collector.

So far, the only way i found is that i can add a prometheus receiver to opentelemetry pipeline which can go and scrap metrics directly from targets. Also the opentelemetry collectors run as a cluster behind load balancer. Does it mean, all the prometheus receivers will scrap the metrics from targets and hence will result in duplication?

Thanks

[rakesh-p]

for anyone using splunk-otel-collector, there is a remote write receiver https://docs.splunk.com/observability/en/gdi/opentelemetry/components/signalfx-gateway-prometheus-remote-write-receiver.html

[JustinMason]

@rakesh-p Im not sure about the signalfx receiver, it may be worth testing. Consider building your own collector using the framework and include this module. OTel Collector Builder

Here is a example of a collector and a initial implementation: https://github.com/JustinMason/otel-collector/tree/main/receiver/prometheusremotewritereceiver

[ArthurSens]

Hi @andrzej-stencel

I'm Arthur, a Prometheus team member and contributor to the otel-prometheus-wg! I've noticed that this component's bootstrap has become stale, so I'm wondering if you're still interested in sponsoring the work here.

I'd love to work on a prototype that focuses on Prometheus Remote-Write v2 since we already know that v1 has a few limitations for good OTLP compatibility.

[dashpole]

I'm also happy to help with the review of a PRW 2.0 receiver

[dashpole]

I talked with @andrzej-stencel. I'm going to take over of the sponsor of the component to unblock development

[andrzej-stencel]

Thank you @dashpole!

[tomatopunk]

Thank you for all. Regarding the sponsor's not to proceed with reviewing and merging the PR, I will no longer be continuing its development.

[ArthurSens]

No worries @tomatopunk! Thanks for quickstarting the conversations here, I'm volunteering to continue your work :)

[adelbot]

I don't understand how can i make test of this new receiver. It seems that is not included in otelcol-contrib version 0.114.0

[dashpole]

@adelbot It is still under development. It will be added to contrib after it reaches alpha.

[aakarshg]

Hi @dashpole any updates on this component btw?

[ArthurSens]

Hi @dashpole any updates on this component btw?

It's slowly advancing, but it's not ready enough to call it alpha yet. We have a few TODOs in the code. Once we solve them, we should be good to publish it.

I don't have permission in this repo, but I was wondering if we could create a nested issue to make contributions easier, is that possible?

[dashpole]

What do you want the nested issue to say? I can create one (or ping me)

[dashpole]

Here you go: https://github.com/open-telemetry/opentelemetry-collector-contrib/issues/37277

[github-actions[bot]]

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

[tpanza]

Any updates or status to share on this?

And what might be an interim workaround to use in the meantime? Most docs on the topic of Prometheus and OpenTelemetry discuss Prometheus being a backend/receiver of OpenTelemetry data, rather than a sender.

[adelbot]

I really want to use OpenTelemtry as a signal transport (logs, metrics, and traces). This transport will send the signals to the appropriate backends. This allows for no/too much vendor lockin. So, my requirement for metrics is to scrape the Prometheus metrics and send them to this transport bus.

What do you think of this vision?

[sfc-gh-chli]

I am looking into this component and it looks like it is only supporting prometheus remote write v2, do we have any plans to support prometheus remote v1 as well?

[dashpole]

@sfc-gh-chli This will only support PRW 2.0. I would be much more challenging to implement support for V1. It was considered many times prior to PRW 2.0, but the lack of atomicity for complex types (histogram, summary) makes it very difficult to properly support the protocol.