Support specifying multiple Namespaces for EMF

[mkielar]

Context

Currently, when configuring EMF Processor, we can only do this:

"logs": {
  "metrics_collected": {
    "prometheus": {
      "log_group_name": "myLogGroup",
      "prometheus_config_path": "env:PROMETHEUS_CONFIG_CONTENT",
      "emf_processor": {
        "metric_namespace": "prod/Namespace"
        "metric_declaration": [
          {
            source_labels    = ["job1"]
            label_matcher    = ...
            dimensions       = ...
            metric_selectors = ...
          },
          {
            source_labels    = ["job2"]
            label_matcher    = ...
            dimensions       = ...
            metric_selectors = ...
          },
        ]
      }
    }
  }
}

As a result, metrics from both prometheus scrapper jobs will be stored in the same Cloudwatch Namespace.

Feature request

It would be nice to have the possibility to specify the metric_namespace inside the metric_declaration JSON, to allow metrics scrapped by different jobs to go to different namespaces.

Justification / use-case

We're running ECS/Fargate services with AppMesh. The ECS Task consists of:

• Service container
• Envoy container
• Cloudwatch Agent Sidecar
• X-Ray Sidecar

I'd like to configure Cloudwatch Agent to send Prometheus metrics published by Envoy to prod/AppMesh Namespace, so that I have all AppMesh-related metrics in one place, and can then use Cloudwatch SEARCH feature to group them on a single Dashboard Widget.

At the same time, however, the Service container may expose its own Prometheus metrics, and these should go to another Namespace (most likely named after the Service itself). This is currently not possible.

[mkielar]

Proposed syntax:

"logs": {
  "metrics_collected": {
    "prometheus": {
      "log_group_name": "myLogGroup",
      "prometheus_config_path": "env:PROMETHEUS_CONFIG_CONTENT",
      "emf_processor": {
        "metric_namespace": "prod/Namespace",      // <=== This could still serve as default
        "metric_declaration": [
          {
            "source_labels": ["envoy"]
            "metric_namespace": "prod/AppMesh",    // <=== Optional override
            "label_matcher":" ...
            "dimensions": ...
            "metric_selectors": ...
          },
          {
            "source_labels": ["service"],
            "metric_namespace": "prod/MyService",  // <=== Optional override
            "label_matcher": ...
            "dimensions": ...
            "metric_selectors": ...
          },
        ]
      }
    }
  }
}

[khanhntd]

Hi @mkielar, thanks for bringing up the issues and I personally think that this feature would benefit a lot for our customer base. Therefore, I will bring up this issue with my team and update in the following date.

[emersonclaireiaquinto]

Any news @khanhntd? My team is in the process of mvoing over from Grafana to Cloudwatch this is would definitely be a nice feature.

[khanhntd]

Hi @Whatforit, sad news is that I am not a part of Amazon anymore. Therefore, my insights won't contribute to CWA's future path anymore. However, I have created an internal ticket for it so CWA's PM will take some time to look at it. Otherwise, if you have a rush, please contact with cloudwatch agent team members.