elasticsearch_exporter icon indicating copy to clipboard operation
elasticsearch_exporter copied to clipboard

Fix elasticsearch v8 cluster_settings unmarshal error

Open takahisa opened this issue 1 year ago • 2 comments

Closes #840

For example, the result of calling the _cluster/settings API on an off-the-shelf Elasticsearch might look like the following.

GET _cluster/settings?include_defaults | jq '.defaults.cluster.routing.allocation.disk'
{
  "threshold_enabled": "true",
  "watermark": {
    "flood_stage.frozen.max_headroom": "20GB",
    "flood_stage": "95%",
    "high": "90%",
    "low": "85%",
    "enable_for_single_data_node": "false",
    "flood_stage.frozen": "95%"
  },
  "include_relocations": "true",
  "reroute_interval": "60s"
}

When a custom setting related to cluster.routing.allocation.watermark.flood_stage is introduced in transient, the output changes as follows.

GET _cluster/settings?include_defaults | jq '.defaults.cluster.routing.allocation.disk'
{
  "threshold_enabled": "true",
  "watermark": {
    "flood_stage": {
      "frozen": "95%",
      "frozen.max_headroom": "20GB"
    },
    "high": "90%",
    "low": "85%",
    "enable_for_single_data_node": "false"
  },
  "include_relocations": "true",
  "reroute_interval": "60s"
}

As can be seen from this example, the JSON returned by the Elasticsearch _cluster/settings API is not uniform. Therefore, workarounds are necessary to unmarshal it into Go structures. This issue is also pointed out in #509.

This PR addresses the above issue in a general way. When unmarshalling JSON into Go structures, the JSON is converted into a flattened format. The flattened map (i.e., key-value pairs) is then remapped to Go structures using mitchellh/mapstructure. By doing this, it ensures that even non-uniform JSON can be unmarshalled reliably.

takahisa avatar Jul 29 '24 15:07 takahisa

I don't have time to do a full review at this moment, but I am always hesitant of adding more dependencies. Particularly in this case, the mapstructure project has been archived and no longer supported: https://github.com/mitchellh/mapstructure.

At a minimum, this would need to be a supported project for a dependency. I'll try to circle back to give a full review soon.

sysadmind avatar Aug 04 '24 15:08 sysadmind

@sysadmind Thank you for your comment. I've got it. I would appreciate it if you could leave a review when you have time.

Particularly in this case, the mapstructure project has been archived and no longer supported: https://github.com/mitchellh/mapstructure.

Oh, sorry. Since the use of mapstructure is not an essential part, it can be removed. I will try removing the dependency on mapstructure and creating the PR again.

takahisa avatar Aug 05 '24 01:08 takahisa

I came accross #840 in last version of the exporter, any idea when this fix could be available ?

sdevineau avatar Nov 19 '24 03:11 sdevineau