VictoriaMetrics
VictoriaMetrics copied to clipboard
Adds firehose http endpoint protocol integration for opentelemetry ingestion.
Is your feature request related to a problem? Please describe
It's possible to forward AWS CloudWatch metrics to the VictoriaMetrics with Firehose http endpoint opentelemetry format. https://aws.amazon.com/blogs/aws/cloudwatch-metric-streams-send-aws-metrics-to-partners-and-to-your-apps-in-real-time/
But it has the following issues:
- Firehose expects strict format for HTTP responses https://docs.aws.amazon.com/firehose/latest/dev/httpdeliveryrequestresponse.html#responseformat
- metric names has prometheus incompatible format
- 1.0 opentemetry format parses incorrectly (attributes have nested maps). https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/CloudWatch-metric-streams-formats-opentelemetry-100.html
Describe the solution you'd like
- Fix compatibility with firehose.
- Introduce new flag for metric name conversion into prometheus compatible format.
- properly parse nested maps for attributes.
Describe alternatives you've considered
No response
Additional information
No response
FYI, the compatibility with AWS Firehose should be fixed in the commit 509df44d03631a863fab00e202a6449c19d784e1
FYI, the -opentelemetry.usePrometheusNaming flag has been introduced in the commit fb42380ef374c394053b4cc4f2b083ed4afc7d09 . If this flag is set, then VictoriaMetrics automatically converts metric names and label names into Prometheus-compatible names for samples ingested via OpenTelemetry format.
FYI, vmagent and single-node VictoriaMetrics properly return responses to Amazon Firehose starting from v1.100.0. They also support the -opentelemetry.usePrometheusNaming command-line flag starting from v1.100.0 release.
@f41gh7 we're converting otel metrics in the same manner as it's done in collector, they are also storing array and map attributes as JSON