consul icon indicating copy to clipboard operation
consul copied to clipboard

Published 20 hours ago • verified publisher icon hashicorp

Check definitions not reflected in output from agent/health/service/id/:service-id

Open Wnthr opened this issue 1 year ago • 1 comments

Overview of the Issue

After registering a service with checks at the agent/service/register endpoint, the check definition is not reflected when using the agent health/service/id/:service-id endpoint.

Reproduction Steps

Steps to reproduce this issue, eg:

  1. Start a consul agent
  2. Register a service (example payload below)
  3. Query that service
# cat pulp-lb.json
{
  "name": "pulp",
  "id": "pulp-lb",
  "tags": ["lb"],
  "tagged_addresses": {
    "lan_ipv4": {
      "address": "10.0.0.100",
      "port": 8080
    }
  },
  "checks": [
    {
      "name": "nginx.service",
      "CheckId": "pulp-nginx-service",
      "args": ["/usr/local/bin/check-systemd-service.sh", "nginx"],
      "DeregisterCriticalServiceAfter": "15m",
      "Interval": "15s"
    },
    {
      "name": "consul.service",
      "CheckId": "pulp-consul-service",
      "args": ["/usr/local/bin/check-systemd-service.sh", "consul"],
      "DeregisterCriticalServiceAfter": "15m",
      "interval": "15s"
    },
    {
      "name": "consul-template.service",
      "CheckId": "pulp-consul-template-service",
      "args": ["/usr/local/bin/check-systemd-service.sh", "consul-template"],
      "DeregisterCriticalServiceAfter": "15m",
      "interval": "15s"
    },
    {
      "name": "pulp_tcp_lan",
      "CheckId": "pulp-tcp-lan",
      "TCP": "10.0.0.100:8080",
      "interval": "5s"
    }
  ]
}

Registering service using curl --request PUT --data @pulp-lb.json http://127.0.0.1:8500/v1/agent/service/register?replace-existing-checks=true

Output from http://127.0.0.1:8500/v1/agent/health/service/id/pulp-lb

{
  "AggregatedStatus": "passing",
  "Service": {
    "ID": "pulp-lb",
    "Service": "pulp",
    "Tags": [
      "lb"
    ],
    "Meta": {},
    "Port": 0,
    "Address": "",
    "TaggedAddresses": {
      "lan_ipv4": {
        "Address": "10.0.0.100",
        "Port": 8080
      }
    },
    "Weights": {
      "Passing": 1,
      "Warning": 1
    },
    "EnableTagOverride": false,
    "Datacenter": "dc1"
  },
  "Checks": [
    {
      "Node": "master",
      "CheckID": "pulp-consul-service",
      "Name": "consul.service",
      "Status": "passing",
      "Notes": "",
      "Output": "",
      "ServiceID": "pulp-lb",
      "ServiceName": "pulp",
      "ServiceTags": [
        "lb"
      ],
      "Type": "",
      "ExposedPort": 0,
      "Definition": {
        "Interval": "0s",
        "Timeout": "0s",
        "DeregisterCriticalServiceAfter": "0s",
        "HTTP": "",
        "Header": null,
        "Method": "",
        "Body": "",
        "TLSServerName": "",
        "TLSSkipVerify": false,
        "TCP": "",
        "TCPUseTLS": false,
        "UDP": "",
        "GRPC": "",
        "OSService": "",
        "GRPCUseTLS": false
      },
      "CreateIndex": 0,
      "ModifyIndex": 0
    },
    {
      "Node": "master",
      "CheckID": "pulp-consul-template-service",
      "Name": "consul-template.service",
      "Status": "passing",
      "Notes": "",
      "Output": "",
      "ServiceID": "pulp-lb",
      "ServiceName": "pulp",
      "ServiceTags": [
        "lb"
      ],
      "Type": "",
      "ExposedPort": 0,
      "Definition": {
        "Interval": "0s",
        "Timeout": "0s",
        "DeregisterCriticalServiceAfter": "0s",
        "HTTP": "",
        "Header": null,
        "Method": "",
        "Body": "",
        "TLSServerName": "",
        "TLSSkipVerify": false,
        "TCP": "",
        "TCPUseTLS": false,
        "UDP": "",
        "GRPC": "",
        "OSService": "",
        "GRPCUseTLS": false
      },
      "CreateIndex": 0,
      "ModifyIndex": 0
    },
    {
      "Node": "master",
      "CheckID": "pulp-nginx-service",
      "Name": "nginx.service",
      "Status": "passing",
      "Notes": "",
      "Output": "",
      "ServiceID": "pulp-lb",
      "ServiceName": "pulp",
      "ServiceTags": [
        "lb"
      ],
      "Type": "",
      "ExposedPort": 0,
      "Definition": {
        "Interval": "0s",
        "Timeout": "0s",
        "DeregisterCriticalServiceAfter": "0s",
        "HTTP": "",
        "Header": null,
        "Method": "",
        "Body": "",
        "TLSServerName": "",
        "TLSSkipVerify": false,
        "TCP": "",
        "TCPUseTLS": false,
        "UDP": "",
        "GRPC": "",
        "OSService": "",
        "GRPCUseTLS": false
      },
      "CreateIndex": 0,
      "ModifyIndex": 0
    },
    {
      "Node": "master",
      "CheckID": "pulp-tcp-lan",
      "Name": "pulp_tcp_lan",
      "Status": "passing",
      "Notes": "",
      "Output": "TCP connect 10.0.0.100:8080: Success",
      "ServiceID": "pulp-lb",
      "ServiceName": "pulp",
      "ServiceTags": [
        "lb"
      ],
      "Type": "",
      "ExposedPort": 0,
      "Definition": {
        "Interval": "0s",
        "Timeout": "0s",
        "DeregisterCriticalServiceAfter": "0s",
        "HTTP": "",
        "Header": null,
        "Method": "",
        "Body": "",
        "TLSServerName": "",
        "TLSSkipVerify": false,
        "TCP": "",
        "TCPUseTLS": false,
        "UDP": "",
        "GRPC": "",
        "OSService": "",
        "GRPCUseTLS": false
      },
      "CreateIndex": 0,
      "ModifyIndex": 0
    }
  ]
}

Consul info for Client (no Server in test setup)

Client info 
# consul info
agent:
        check_monitors = 6
        check_ttls = 0
        checks = 10
        services = 4
build:
        prerelease =
        revision = 7736539d
        version = 1.17.2
        version_metadata =
consul:
        acl = disabled
        bootstrap = true
        known_datacenters = 1
        leader = true
        leader_addr = 10.0.0.100:8300
        server = true
raft:
        applied_index = 14527
        commit_index = 14527
        fsm_pending = 0
        last_contact = 0
        last_log_index = 14527
        last_log_term = 10
        last_snapshot_index = 0
        last_snapshot_term = 0
        latest_configuration = [{Suffrage:Voter ID:25010851-3a85-32f0-a6d0-fcf94dd3f0e5 Address:10.0.0.100:8300}]
        latest_configuration_index = 0
        num_peers = 0
        protocol_version = 3
        protocol_version_max = 3
        protocol_version_min = 0
        snapshot_version_max = 1
        snapshot_version_min = 0
        state = Leader
        term = 10
runtime:
        arch = amd64
        cpu_count = 2
        goroutines = 223
        max_procs = 2
        os = linux
        version = go1.21.6
serf_lan:
        coordinate_resets = 0
        encrypted = false
        event_queue = 1
        event_time = 10
        failed = 0
        health_score = 0
        intent_queue = 1
        left = 0
        member_time = 10
        members = 1
        query_queue = 0
        query_time = 1
serf_wan:
        coordinate_resets = 0
        encrypted = false
        event_queue = 0
        event_time = 1
        failed = 0
        health_score = 0
        intent_queue = 0
        left = 0
        member_time = 1
        members = 1
        query_queue = 0
        query_time = 1

Client config JSON

{
  "retry_join" : ["10.0.0.100"],
  "data_dir": "/opt/consul/data",
  "log_level": "INFO",
  "server": true,
  "node_name": "master",
  "addresses": {
    "https": "10.0.0.100"
  },
  "bind_addr": "10.0.0.100",
  "ui": true,
  "bootstrap_expect": 1
}

Operating system and Environment details

NAME="AlmaLinux"
VERSION="9.3 (Shamrock Pampas Cat)"
ID="almalinux"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.3"
PLATFORM_ID="platform:el9"
PRETTY_NAME="AlmaLinux 9.3 (Shamrock Pampas Cat)"
ANSI_COLOR="0;34"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:almalinux:almalinux:9::baseos"
HOME_URL="https://almalinux.org/"
DOCUMENTATION_URL="https://wiki.almalinux.org/"
BUG_REPORT_URL="https://bugs.almalinux.org/"

ALMALINUX_MANTISBT_PROJECT="AlmaLinux-9"
ALMALINUX_MANTISBT_PROJECT_VERSION="9.3"
REDHAT_SUPPORT_PRODUCT="AlmaLinux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.3"

Log Fragments

# journalctl -fu consul
Jan 30 21:48:23 social-bonefish consul[1117]: 2024-01-30T21:48:23.606Z [INFO]  agent.http: Request cancelled: method=GET url="/v1/health/service/pulp?index=14501&passing=1&stale=&tag=backend&wait=60000ms" from=127.0.0.1:40846 error="context canceled"
Jan 30 21:48:23 social-bonefish consul[1117]: 2024-01-30T21:48:23.607Z [INFO]  agent.http: Request cancelled: method=GET url="/v1/health/service/vault?index=14509&passing=1&stale=&tag=backend&wait=60000ms" from=127.0.0.1:40848 error="context canceled"
Jan 30 21:48:23 social-bonefish consul[1117]: 2024-01-30T21:48:23.607Z [INFO]  agent.http: Request cancelled: method=GET url="/v1/health/service/pulp?index=14501&passing=1&stale=&tag=lb&wait=60000ms" from=127.0.0.1:40860 error="context canceled"
Jan 30 21:50:18 social-bonefish consul[1117]: 2024-01-30T21:50:18.544Z [INFO]  agent: Synced check: check=service:pulp-backend:1
Jan 30 21:51:31 social-bonefish consul[1117]: 2024-01-30T21:51:31.494Z [INFO]  agent: Synced check: check=service:vault-backend:1
Jan 30 21:52:47 social-bonefish consul[1117]: 2024-01-30T21:52:47.808Z [INFO]  agent: Synced check: check=service:vault-lb:1
Jan 30 21:53:27 social-bonefish consul[1117]: 2024-01-30T21:53:27.593Z [INFO]  agent.http: Request cancelled: method=GET url="/v1/health/service/pulp?index=14518&passing=1&stale=&tag=backend&wait=60000ms" from=127.0.0.1:35612 error="context canceled"
Jan 30 21:53:27 social-bonefish consul[1117]: 2024-01-30T21:53:27.594Z [INFO]  agent.http: Request cancelled: method=GET url="/v1/health/service/vault?index=14528&passing=1&stale=&tag=backend&wait=60000ms" from=127.0.0.1:35620 error="context canceled"
Jan 30 21:53:27 social-bonefish consul[1117]: 2024-01-30T21:53:27.595Z [INFO]  agent.http: Request cancelled: method=GET url="/v1/health/service/vault?index=14528&passing=1&stale=&tag=lb&wait=60000ms" from=127.0.0.1:35616 error="context canceled"
Jan 30 21:53:27 social-bonefish consul[1117]: 2024-01-30T21:53:27.597Z [INFO]  agent.http: Request cancelled: method=GET url="/v1/health/service/pulp?index=14518&passing=1&stale=&tag=lb&wait=60000ms" from=127.0.0.1:35592 error="context canceled"

Wnthr avatar Jan 30 '24 22:01 Wnthr

@Wnthr , looking into this. The check definitions - DeregisterCriticalServiceAfter, interval, TCP are in output with default values, actual values provided during service registration are not displayed in output.

vijayraghav-io avatar Jan 31 '24 05:01 vijayraghav-io