mysqld_exporter icon indicating copy to clipboard operation
mysqld_exporter copied to clipboard

add metric displaying whether replication is configured or not

Open mmiller1 opened this issue 3 years ago • 7 comments

This metric would be useful in the event that after a crash (or just as a result of human error) the replication status is reset. Currently if the replication configuration is removed, all metrics related to replication disappear making it cumbersome to alert on. I considered using the absent() functions in prometheus to detect this, but that would depend on creating alert definitions for each host that you expect the metrics to exist for, which is not very practical.

mmiller1 avatar Dec 03 '21 16:12 mmiller1

You don't need a new metric for this, you can use any existing ones mysql_slave_status_*. For example

absent(mysql_slave_status_connect_retry{instance="foobar"})

returns 1 if slave is not configured or nothing if it is.

roman-vynar avatar Dec 08 '21 20:12 roman-vynar

As I mentioned, absent() is not ideal here because we have hundreds of mysql hosts that we need to alert on in the event of replication misconfiguration, and defining an alert for instance="foo001" through instance="foo999" is not practical. With this metric all we need to do is define a single alert: mysql_slave_status_is_configured == 0.

When replication is not configured at all (after a reset slave command, for example) none of the mysql_slave_status_* metrics exist prior to this PR.

mmiller1 avatar Dec 08 '21 21:12 mmiller1

All you need is a regex to match all your mysql hosts {instance=~"foo\d{3}"} or some any other label you may want to tag your mysql instances to be captured.

roman-vynar avatar Dec 09 '21 09:12 roman-vynar

Sorry, no, this doesn't work. in order for absent() to return one on a regex match, ALL hosts that match the regex must be missing the metric. I need to know if any single host is missing the metric.

mmiller1 avatar Dec 09 '21 13:12 mmiller1