Nomad Service Catalog should have a filter or metadata for health status

[mr-karan]

Proposal

Currently, I am querying the list of Nomad Services using consul-template Nomad functions. I couldn't find a way to filter only the list of healthy services.

However, when I query the list of nomad services for this namespace, I can find these services here and it doesn't have any metadata whether the healthcheck failed or not:

[
    {
        "Address": "192.168.29.76",
        "AllocID": "4565fd09-ae36-06c7-229a-bd685ea5b8f4",
        "CreateIndex": 2510,
        "Datacenter": "dc1",
        "ID": "_nomad-task-4565fd09-ae36-06c7-229a-bd685ea5b8f4-web-doggo-web-http",
        "JobID": "doggo",
        "ModifyIndex": 2510,
        "Namespace": "default",
        "NodeID": "987413db-db5f-0267-caff-cc82086234f0",
        "Port": 27159,
        "ServiceName": "doggo-web",
        "Tags": [
            "doggo",
            "web"
        ]
    },
    {
        "Address": "192.168.29.76",
        "AllocID": "52efad7f-6b99-de8f-50b3-9e4358ea7c3c",
        "CreateIndex": 2477,
        "Datacenter": "dc1",
        "ID": "_nomad-task-52efad7f-6b99-de8f-50b3-9e4358ea7c3c-web-doggo-web-http",
        "JobID": "doggo",
        "ModifyIndex": 2477,
        "Namespace": "default",
        "NodeID": "987413db-db5f-0267-caff-cc82086234f0",
        "Port": 28421,
        "ServiceName": "doggo-web",
        "Tags": [
            "doggo",
            "web"
        ]
    },
    {
        "Address": "192.168.29.76",
        "AllocID": "fed569fb-3dac-8ed5-2c0c-7216c0a00ae9",
        "CreateIndex": 2524,
        "Datacenter": "dc1",
        "ID": "_nomad-task-fed569fb-3dac-8ed5-2c0c-7216c0a00ae9-web-doggo-web-http",
        "JobID": "doggo",
        "ModifyIndex": 2524,
        "Namespace": "default",
        "NodeID": "987413db-db5f-0267-caff-cc82086234f0",
        "Port": 24021,
        "ServiceName": "doggo-web",
        "Tags": [
            "doggo",
            "web"
        ]
    }
]

Use-cases

When using NGINX or similar proxy, it's useful to only filter the upstream servers which are healthy. This will help these proxies to not send the request to the upstream servers where health checks are failing.

Attempted Solutions

Couldn't find a workaround

[jrasell]

Hi @mr-karan and thanks for raising, what I agree, would be a useful addition to the Nomad service discovery feature. To provide context, this is a known and designed current limitation as the health check state is only stored on the Nomad client that is performing the checking.

[devminded]

Related to https://github.com/prometheus/prometheus/issues/11775

[konnextv]

Nomad newbie here, this peculiarity should definitely be mentioned in the template docs. I thought it is obvious that only healthy service instances are included in nomadService and wasted hours searching the cause of my problems.