Make it easier to customize node selector

[minrk]

node queries are mostly selected with node names, e.g. node=~".*user.*". This often works, but I wonder if making this customizable is feasible? E.g. specifying the node label and value pattern in some variable?

For instance, I could imagine the node usage metrics using sum(metric) by (label_cloud_google_com_gke_nodepool) to automatically get metrics by node pool or some such.

[yuvipanda]

@minrk yeah, I started moving everything to using labels in cluster.jsonnet, it should be the case everywhere.

[yuvipanda]

Need to make sure it works for AWS as well though.

[minrk]

yeah, I don't have a good grasp on how variables work (whether in jsonnet or grafana) to know, but it seems we could specify a "node pool node label" in a variable and use that. I don't know if that needs to be a template variable as opposed to a grafana one, though

[yuvipanda]

I think of these as 'build time' changes, so we can probably make the deployer try to detect what labels are present somehow and determine what the queries should be?