consul_exporter
consul_exporter copied to clipboard
prometheus
Add node metadata and service tags as metrics.
As discussed in #45, I've added service tags as a separate metric. One tag per series.
I've done the same with node metadata. One key/value pair per series.
@rbestbmj Can you rebase on top of current master please?
@brian-brazil Using key and value labels to wrap the node metadata avoids the issues of varying labels across timeseries. It also makes it harder to match multiple labels at once though. Do you think that's a reasonable approach?
Do you think that's a reasonable approach?
No, everywhere else this come up we use labels directly. Watch out for varying labels, you'll need to ensure that every mentioned label is on every use of the metric.
@rbestbmj It doesn't look like a correct rebase, your pull request now includes a bunch of other commits as well.
Furthermore, we'll want to change the implementation and export one time series per node, including all its metadata as labels. The label name will need to be sanitized and we need to workaround the restrictions in client_golang. I'm happy to provide more directions if you're interested in working on that.
@grobie Sorry, rebase sorted. I'm also more than happy to keep working on the feature.
@brian-brazil @grobie Just to clarify, you're suggesting that each series should include all the metadata as labels, i.e:
consul_node_meta_data{environment="dev",project="foo",component="bar"}?
It's quite likely that metadata could vary between nodes, meaning you'd have inconsistent labels between series. I guess this requires whoever is maintaining the consul cluster and the exporter to ensure consistent metadata keys but that seems a bit restrictive to me.
What's the objection to using key and value labels?
When I looked at this briefly back in October it looked to me like there wasn't a means of dynamically generating the set of labels for a metric based on the metadata we receive back from the API. Would you suggest, then, that we provide the keys we want the exporter to use ahead of time (via an argument?)? Or are these the restrictions in client_golang you mentioned?
@rbestbmj Correct, that's how meta data has been modeled so far in other exporters. For consistency, we'd want to not introduce a different approach here. It makes it also easier to match and return all metalabels at once.
To deal with varying labels between nodes, the general recommendation is to keep them at sync in the source (e.g. Consul). Though, in reality that won't always be the case as you have pointed out. In Kubernetes related exporters all missing labels are exported with an empty value to ensure a consistent set being exported. I believe @beorn7 was considering adding support for that to client_golang to make it easier.
@grobie Even if you export missing labels with an empty value, that implies that the exporter knows which metadata keys it will expect to find. So, would you suggest we pass the node metadata keys we're interested in to the exporter with a flag or otherwise?
