Automatically discover all existing metrics

[OElGebaly]

Hi everyone .. i hope that i am posting in the right topic ...

i am asking if there is a way to configure scrapping all the existing metrics for a specific namespace without specifying the "metricname" or "dimensions"

[kaihendry]

I too am looking for a simpler solution that explicitly naming every aws_metric_name I want under a aws_namespace...

[matthiasr]

I don't think this is possible at the moment, but I see the value in a simple "give me everything" option for simple situations. It should probably be opt-in, given the potential costs. What could the configuration for this look like?

[matthiasr]

See also #351 for a more specific request for automatic discovery.

[dragoangel]

Hi, what if allow some sort of tempting the configs in foreach way or using wildcards.

Wildcard options:

1. give me all metrics in Namespace X and use for all of them Dimension Y:

- aws_namespace: AWS/NetworkELB
  aws_metric_name: *
  aws_dimensions: [AvailabilityZone, LoadBalancer, TargetGroup]
  aws_statistics: [Average]

2. give me all metrics in Namespace X* with name Y:

- aws_namespace: Project-*
  aws_metric_name: Info-Logs-Counter
  aws_statistics: [Sum]

Foreach: Imagine we have:

- var_name: Environment
  var_values: [SBX, DEV, STG, PRD]

- var_name: Service
  var_values: [Identity, DataProvider, Media]

- var_name: LogLevel
  var_values: [Info, Warning, Error]

And we need write how much code to describe all this? 🤔 Now I will use bash to generate my config I think... But this create unreadable configs, which impossible to maintain and modify. But we can do this without scripting 🤣, here vars come in play:

- aws_namespace: MyService-${Environment}-${Service}
  aws_metric_name: ${LogLevel}-Logs-Counter
  aws_statistics: [Sum]

And we have clear and readable config

[matthiasr]

Some only vaguely connected thoughts on this:

We should probably not default to scraping all namespaces due to cost. How common is it to have an open set of namespaces?

On the other hand, aws_metric_name and aws_dimensions could have sensible default handling if unspecified. The combination of aws_metric_name: * and fixed dimensions and statistics seems perilous, this will be invalid in too many situations, for example you don't want to average the request latency.

When would discovery happen? Is it enough to do this at exporter startup? Or even in an offline mapping generator?

I am not a fan logic-in-YAML. If you need this level of customization I'd rather use a programming language and dump out the YAML (or JSON). We do this a lot, usually running the generator in deploy pipelines. It lets you pull in information from wherever, and use any logic on it, without having to build half a scripting language into every tool.

[matthiasr]

(removing the help wanted label until we know what exactly we want)

[matthiasr]

Another, more fundamental question, is how a cloudwatch_exporter with automatic discovery would relate to YACE. I don't particularly want to expend effort solving the same use case twice, unless there is a compelling reason they cannot be reconciled.

[djloude]

Not a great solution but a while ago I created a GitLab pipeline that uses an ansible playbook to setup prometheus, grafana, cloudwatch_exporter from scratch for multiple accounts. Part of the exporter setup was a few scripts that grabs all aws metrics via namespace and passes it to an ansible template to create the cloudwatch.yml file. I quickly modified it some to be ran locally and without all the extra stuff. You can check it out here.

[matthiasr]

@djloude that's neat! Do you want to submit a PR to add a link to the readme?

[omri-shilton]

any updates on this? we would also like to have all the metrics from a specific namespace, ive tried using the builder tool, but its outdated and doesnt seem to work properly

[matthiasr]

I'm going to close this issue because I believe fully automatic discovery is out of scope for this exporter. If you want something like it, please submit fixes to @djloude's config builder, or use YACE (I won't be mad, I promise!)