Introduce a way to tag clusters (and, optionally, nodes)

[michaelklishin]

Per discussion with @SimonUnge but also @stefanmoser @mkuratczyk.

Problem Definition

Larger users can have thousands of clusters used for all kinds of purposes. Sometimes it may be necessary to perform certain operations (e.g. upgrades) on a subset of them, e.g. those that are used by development environments only.

Right now there only so many ways of doing it:

1. Keep this cluster metadata in an external store. This is not always an option
2. Prefix cluster name with production-* or something like that. This is fragile and allows for only a couple of tags

Cluster Tags

One solution can be configuring cluster metadata with a map (a set of pairs) in rabbitmq.con:

cluster_tags.environment = production

cluster_tags.region = us-east
cluster_tags.az = us-east-3

This map can be then added to GET /api/overview HTTP API responses and using a new object in Prometheus output:

rabbitmq_identity_cluster_tags{environment = "production", region = "us-east", az = "us-east-3"}

This data can be stored in the node's application environment or in a global runtime parameter (for "last write wins" consistency).

Node Tags

The same idea can be extended to node tags that will only be stored in the application environment since this data is local to the node.

node_tags.environment = production

node_tags.region = us-east
node_tags.az = us-east-3

rabbitmq_identity_node_tags{environment = "production", region = "us-east", az = "us-east-3"}

The usefulness of this is less clear.