Feature request - Watermark and read_only indices

[frenkye]

Hi,

great exporter, but Iam missing variable for watermark and percent usage of node.

1. Node: curl localhost:9200/_cat/allocation?h=node,disk.percent
2. Watermark: curl localhost:9200/_cluster/settings if not present in persistent or transient settings, then 85% is defaut.

More about watermark is here https://www.elastic.co/guide/en/elasticsearch/reference/7.5/disk-allocator.html

Because if ES hit the flood_stage (introduce in version 6, before high was the limit), then all indices are switch to read only and have to be manually unlock AFAIK, they planned to do auto unlocking in disk space is freed, but don't know if that is implemented in new versions.

With this would be also helpfull if there was variable for tracking number of read only indices, this can be found under /_all/_settings if set index.blocks.read_only_allow_delete: true

For example like was issue here https://github.com/elastic/kibana/issues/13685

[stefnestor]

Because if ES hit the flood_stage (introduce in version 6, before high was the limit), then all indices are switch to read only and have to be manually unlock AFAIK, they planned to do auto unlocking in disk space is freed, but don't know if that is implemented in new versions.

Confirming from Elastic's flood-stage troubleshooting guide, the read_only_allow_delete block automatically removes after node leaves flood-stage+high watermark thresholds. Implemented v7.4.0 via elasticsearch#42559.

[sysadmind]

As for the original request, the --collector.clustersettings argument exports cluster settings metrics including watermark settings. i.e.

# HELP elasticsearch_clustersettings_allocation_watermark_flood_stage_ratio Flood stage watermark as a ratio.
# TYPE elasticsearch_clustersettings_allocation_watermark_flood_stage_ratio gauge
elasticsearch_clustersettings_allocation_watermark_flood_stage_ratio 0.95
# HELP elasticsearch_clustersettings_allocation_watermark_high_ratio High watermark for disk usage as a ratio.
# TYPE elasticsearch_clustersettings_allocation_watermark_high_ratio gauge
elasticsearch_clustersettings_allocation_watermark_high_ratio 0.9
# HELP elasticsearch_clustersettings_allocation_watermark_low_ratio Low watermark for disk usage as a ratio.
# TYPE elasticsearch_clustersettings_allocation_watermark_low_ratio gauge
elasticsearch_clustersettings_allocation_watermark_low_ratio 0.85

I believe that the percentage metric may be missing, however it can be calculated from elasticsearch_filesystem_data_size_bytes and elasticsearch_filesystem_data_available_bytes