grafana-dashboard-kubernetes-prometheus
grafana-dashboard-kubernetes-prometheus copied to clipboard
New namespace template
- Setup namespace templating
- Fix device naming to accept devices like /dev/xvda9 (based on http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/device_naming.html)
@gianrubio, thank you for contribution!
Not sure about device=~\"^/dev/(sd|hd|xvd).*$\"
, as it will allow other partitions (different from 9) to be included.
I decided to use /dev/[sf]da9
strictly because of CoreOS Disk Layout, as far as I'm using CoreOS.
Would you be so kind to comment this moment more wide? What is your opinion?
@tyranron you're absolutely right about the partition 9. However you can mount devices as /dev/xvd. Look on my current cluster
What do you think to use /dev/([sf]|xv)da9 ?
What do you think to use /dev/([sf]|xv)da9 ?
@gianrubio 👍👍👍
@gianrubio I've dived into it a bit closely and found some problems:
- Unlike Node templating, the
namespace
label is not present on every metric. Actually, it is present only on k8s containers. So I removed Namespace templating from queries not related to k8s. -
cAdvisor's container abstraction is based on lmctfy's so containers are inherently nested hierarchically.
That means that containers metrics withid="/"
show total system usage, and we are using that in our "Total Usage" stats. Those metrics do not havenamespace
label. So for templating by Namespace in "Total Usage" stats we need to sum per-container usage now, likeid=~"^/[^/]+/[^/]+$"
. It's okay when we choose concrete Namespace, but numbers differ when we chooseNamespace: All
, because now queries do not collect system usage that is not related to containers:
Ideally, we need to use# This is not 0 sum(container_fs_usage_bytes{device=~"^/dev/([sf]|xv)da9$",id="/"}) - sum(container_fs_usage_bytes{device=~"^/dev/([sf]|xv)da9$",id=~"^/[^/]+/[^/]+$"})
id="/"
forNamespace: All
, andid=~"^/[^/]+/[^/]+$"
in other cases. But Grafana templating does not provide such capabilities.
I think we need to customize Prometheusrelabel_config
a bit more here.