[metrics]: Metrics are no longer pulled when using aws_tag_select

[lukayeh]

Context information

• AWS service: AmazonMQ & Amazon MSK
• CloudWatch namespace: AWS/AmazonMQ & AWS/Kafka
• Link to metrics documentation for this service: n/a
• AWS region of the exporter: eu-central-1
• AWS region of the service: eu-central-1

When running Cloudwatch exporter with aws_tag_select enabled for services like RDS and Elasticache I get metrics returned, however when doing the same for MSK and MQ it does not return any metrics.

We know this works

    region: eu-central-1
    period_seconds: 60
    metrics:
    - aws_namespace: AWS/AmazonMQ
      aws_metric_name: MessageCount
      aws_dimensions:
        - Broker
      aws_statistics:
        - Sum

This does not:

    metrics:
    - aws_namespace: AWS/AmazonMQ
      aws_metric_name: MessageCount
      aws_dimensions:
        - Broker
      aws_statistics:
        - Sum
	  aws_tag_select:
	    tag_selections: 
	      Project: []
	    resource_type_selection: "mq:broker"
	    resource_id_dimension: [Broker]

This also does not work ❌:

    region: eu-central-1
    period_seconds: 60
    metrics:
    - aws_namespace: AWS/AmazonMQ
      aws_metric_name: MessageCount
      aws_dimensions:
        - Broker
      aws_statistics:
        - Sum
      aws_tag_select:
        tag_selections:
          Project: ["legit_tag"]
        resource_type_selection: "mq:broker"
        resource_id_dimension: "Broker"

For Kafka I've tried this but this also refuses to retrieve metrics:

  - aws_namespace: AWS/Kafka
    aws_metric_name: CpuCreditUsage
    aws_dimensions:
      - Cluster Name
    aws_statistics:
      - Average
    aws_tag_select:
      tag_selections: 
          Project: []
      resource_type_selection: "kafka:cluster"
      resource_id_dimension: "Cluster Name"

Exporter logs

WARNING: (listDimensions) ignoring metric AWS/AmazonMQ:MessageCount due to dimensions mismatch

What do you expect to happen?

Metrics return when using aws_tag_select

What happened instead?

No metrics are returned due to dimensions mismatch

[matthiasr]

The "usual suspect" when aws_tag_select doesn't work is a mismatch between the ARN format (which the exporter gets back when asking for tags) and the resource ID dimension. According to this page, the broker ARN ends in broker:MyBroker:b-1234a5b6-78cd-901e-2fgh-3i45j6k178l9 while the exporter, by default, looks for anything after the last :. I am guessing that in this case, the Broker dimension would be MyBroker, so maybe try something like arn_resource_id_regexp: 'broker:([^:]*):[^:]*$'?

[matthiasr]

For MSK, I can't really find a page about the ARN format, but this page has some examples that suggest a similar problem.

For both, once you get it to work, please submit a PR for the examples.

[Kalmonipa]

Using arn_resource_id_regexp: 'broker:([^:]*):[^:]*$' doesn't seem to work...

The brokers have a -1 or -2 suffix because they're active/standby instances so the resource name in the ARN doesn't quite match the Broker dimension in Cloudwatch. AWS Docs for reference An example is an ARN like: broker:test-broker:b-1234a5b6-78cd-901e-2fgh-3i45j6k178l9 and this is both instances of the broker represented in Cloudwatch:

 aws cloudwatch list-metrics --namespace AWS/AmazonMQ --metric-name NetworkIn
{
    "Metrics": [
        {
            "Namespace": "AWS/AmazonMQ",
            "MetricName": "NetworkIn",
            "Dimensions": [
                {
                    "Name": "Broker",
                    "Value": "test-broker-1"
                }
            ]
        },
        {
            "Namespace": "AWS/AmazonMQ",
            "MetricName": "NetworkIn",
            "Dimensions": [
                {
                    "Name": "Broker",
                    "Value": "test-broker-2"
                }
            ]
        }
    ]
}

Any thoughts on how to work around the suffixes?