VictoriaMetrics icon indicating copy to clipboard operation
VictoriaMetrics copied to clipboard

Adds firehose http endpoint protocol integration for opentelemetry ingestion.

Open f41gh7 opened this issue 1 year ago • 4 comments

Is your feature request related to a problem? Please describe

It's possible to forward AWS CloudWatch metrics to the VictoriaMetrics with Firehose http endpoint opentelemetry format. https://aws.amazon.com/blogs/aws/cloudwatch-metric-streams-send-aws-metrics-to-partners-and-to-your-apps-in-real-time/

But it has the following issues:

  • Firehose expects strict format for HTTP responses https://docs.aws.amazon.com/firehose/latest/dev/httpdeliveryrequestresponse.html#responseformat
  • metric names has prometheus incompatible format
  • 1.0 opentemetry format parses incorrectly (attributes have nested maps). https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-formats-opentelemetry-100.html

Describe the solution you'd like

  • Fix compatibility with firehose.
  • Introduce new flag for metric name conversion into prometheus compatible format.
  • properly parse nested maps for attributes.

Describe alternatives you've considered

No response

Additional information

No response

f41gh7 avatar Mar 28 '24 11:03 f41gh7

FYI, the compatibility with AWS Firehose should be fixed in the commit 509df44d03631a863fab00e202a6449c19d784e1

valyala avatar Apr 02 '24 22:04 valyala

FYI, the -opentelemetry.usePrometheusNaming flag has been introduced in the commit fb42380ef374c394053b4cc4f2b083ed4afc7d09 . If this flag is set, then VictoriaMetrics automatically converts metric names and label names into Prometheus-compatible names for samples ingested via OpenTelemetry format.

valyala avatar Apr 02 '24 23:04 valyala

FYI, vmagent and single-node VictoriaMetrics properly return responses to Amazon Firehose starting from v1.100.0. They also support the -opentelemetry.usePrometheusNaming command-line flag starting from v1.100.0 release.

valyala avatar Apr 04 '24 13:04 valyala

@f41gh7 we're converting otel metrics in the same manner as it's done in collector, they are also storing array and map attributes as JSON

AndrewChubatiuk avatar Apr 04 '24 15:04 AndrewChubatiuk