Support Service-Level UDP Check

[lornasong]

trafficstars

Currently ESM supports a node-level UDP check but not a service-level UDP check.

Practitioners may want to have a UDP check to know whether a service on a specific port if healthy and not just overall node health. These external services might be dynamically registered and not just a static list of services. It would therefore be helpful to be able to dynamically register UDP checks similar to how HTTP checks can be registered dynamically via /catalog/register

It would be possible to use a script check feature (not yet built out) to implement service-level UDP checks for a static list of services. Since we want to limit dynamic script checks in order to prevent remote code execution security risk, they are not a good option for dynamically registering UDP checks.

References:

• Use case description: https://github.com/hashicorp/consul-esm/issues/42#issuecomment-611170716
• Details on why existing node-check does not satisfy use case: https://github.com/hashicorp/consul-esm/issues/42#issuecomment-622034756

[NonLogicalDev]

Bumping this, as UDP checks can be useful for testing liveliness of DNS services.

I would further suggest potentially having a pre-configured script checks which can be requested via HTTP API by their "script-id". Such checks could receive service definition and parameters from STDIN.

Example could be:

$ cat <<EOF > /srv/dig_check.sh 

PARAMS=$(cat -)

address=$(echo $PARAMS | jq '.service.address')
port=$(echo $PARAMS | jq '.service.port')
target=$(echo $PARAMS | jq '.params.target')

dig @${address} -p ${port:-53} ${target}
EOF

$ cat <<EOF > /etc/consul.d/registered-checks.json
{
    "check": {
        "script-id": "dig-check",
        "exec": "/srv/dig_check.sh"
    }
}
EOF

$ cat <<EOF > service.json
{
  "service": {
    "name": "dns",
    "port": 53,
    "check": {
        "script-id": "dig-check"
        "params": { "target": "google.com" }
        "interval": "10s"
    }
  }
}
EOF

consul service register service.json

[lornasong]

@NonLogicalDev, thank you for the comment! We’ve just recently moved towards using the reaction feature to gauge community interest. So feel free to also add a :+1: to the description of this issue.

Regarding script checks, you may have already come across it, but in case you haven’t, we have an issue open for script checks here https://github.com/hashicorp/consul-esm/issues/42. If you prefer a script checks implementation, feel free to :+1: in that issue.

Your proposal to pass in the service definition + parameter with all script checks is interesting! It certainly solves the issue of how users can write scripts with variables. You mentioned that pre-configured scripts checks could be requested via HTTP API by their 'script-id'. I’m curious what your thoughts are if they were local to ESM e.g. ESM would load 'registered-checks.json' on start? This is something @cbroglie, who also proposed the idea of having a script-id, suggested earlier in https://github.com/hashicorp/consul-esm/issues/42#issuecomment-611170716.

Thanks!

[NonLogicalDev]

@lornasong Thank you for reply! It does seem like @cbroglie's approach is pretty much the same as mine, with only distinction that if this feature were to be implemented, allowing parameters could open up a treasure trove of opportunities for custom checks.

~~I would say I am not opposed to this being a part of ESM, but it would surely be nice to have a safer more structured alternative to running arbitrary shell script checks out of the box.~~

Update: Am I adding the comment to the right Project? =] Now that I am thinking about this I am a little confused about the relationship between consul-esm and consul agent.

A small expansion on my idea while I am at it:

The parameters could be passed in as a JSON over stdin like I suggested, or probably better if they are passed in as env variables:

The input params:

{ "target": "value", "server": "another_value" }

Would become:

PARAMS_TARGET=shell_escaped("value")
PARAMS_SERVER=shell_escaped("another_value")

And allow the user to specify a mapping of parameters to env variables:

Here are a few variants:

1. Shell Interpolation.

$ cat <<EOF > /etc/consul.d/registered-checks.json
{
    "checks": [
      {
        "script-id": "dig-check",
        "shell": "dig @${DIG_SERVER} \"${DIG_SERVER}\""
        "params": {"target": "DIG_TARGET", "server": "DIG_SERVER"}
      }
    ]
}
EOF

2. Safer Interpolation using go templating.

$ cat <<EOF > /etc/consul.d/registered-checks.json
{
    "checks": [
      {
        "script-id": "dig-check",
        "args": [ "dig", "@{{.DIG_TARGET}}", "{{.DIG_SERVER}}" ]
        "params": {"target": "DIG_TARGET", "server": "DIG_SERVER"}
      }
    ]
}
EOF

3. More elaborate parameter definition, with schema, (though I think that it is safer and easier to just disallow nested dicts and lists in params)

$ cat <<EOF > /etc/consul.d/registered-checks.json
{
    "checks": [
      {
        "script-id": "dig-check",
        "args": [ "dig", "@{{.DIG_TARGET}}", "{{.DIG_SERVER}}" ]
        "params": {
          "target": {
            "env": "DIG_TARGET",
            "type": "string"
          },
          "server": {
            "env": "DIG_SERVER",
            "type": "string"
          }
        }
      }
    ]
}
EOF

[lornasong]

@NonLogicalDev, thanks for the additional ideas on your proposal! I’d be interested to learn more about any use cases you have in mind for using environment variables. Previous script check use-cases I’ve heard are running the same script for multiple checks for one ESM e.g. one ESM runs multiple udp checks for different addresses. Environment variables wouldn’t be as helpful for this case only one address environment variable can be set for a single ESM. So, interested to learn more if you're able to share!

Regarding consul-esm vs. consul, the relationship can be kind of confusing. The service definition (service.json in your initial comment) which declares the check is stored in Consul. ESM retrieves this service definition from Consul and uses the check field information to run the check. In order to support script checks, the implementation will likely require changes to both projects - a change in Consul to store script-specific check information and a change in ESM to retrieve and run the script check.

Hope that clarifies a bit! If not, let me know any questions and if that changes your thoughts on storing the script arguments (registered-checks.json in your example) in ESM. Thanks!

[tristanmorgan]

Cross-linking hashicorp/consul#12722