kong
kong copied to clipboard
Published
20 hours ago •
Kong
Kong
[healthchecks] failed setting peer status: no peer found by name and address
Summary
When I created Upstream then add target to this upstream (target response status 404 in health check path), Kong showed target is healthy and not request a check to Target.
When I ran kong reload, Kong was requested to target then mark it as UNHEALTHY correctly
Error log:
2021/05/18 11:25:24 [warn] 4071345#0: *6523243 [lua] balancer.lua:316: [healthchecks] failed setting peer status (upstream: 5fb68818-d502-41ae-abcc-a41ff1405783:abc): no peer found by name '10.3.6.2' and address 10.3.6.2:31072, context: ngx.timer
2021/05/18 11:25:24 [warn] 4071356#0: *6523248 [lua] balancer.lua:316: [healthchecks] failed setting peer status (upstream: 5fb68818-d502-41ae-abcc-a41ff1405783:abc): no peer found by name '10.3.6.2' and address 10.3.6.2:31072, context: ngx.timer
2021/05/18 11:25:24 [warn] 4071359#0: *6523252 [lua] balancer.lua:316: [healthchecks] failed setting peer status (upstream: 5fb68818-d502-41ae-abcc-a41ff1405783:testupstream): no peer found by name '10.3.6.2' and address 10.3.6.2:31072, context: ngx.timer
2021/05/18 11:25:24 [warn] 4071357#0: *6523254 [lua] balancer.lua:316: [healthchecks] failed setting peer status (upstream: 5fb68818-d502-41ae-abcc-a41ff1405783:testupstream): no peer found by name '10.3.6.2' and address 10.3.6.2:31072, context: ngx.timer
2021/05/18 11:25:24 [warn] 4071354#0: *6523258 [lua] balancer.lua:316: [healthchecks] failed setting peer status (upstream: 5fb68818-d502-41ae-abcc-a41ff1405783:testupstream): no peer found by name '10.3.6.2' and address 10.3.6.2:31072, context: ngx.timer
2021/05/18 11:25:24 [warn] 4071349#0: *6523260 [lua] balancer.lua:316: [healthchecks] failed setting peer status (upstream: 5fb68818-d502-41ae-abcc-a41ff1405783:testupstream): no peer found by name '10.3.6.2' and address 10.3.6.2:31072, context: ngx.timer
2021/05/18 11:25:24 [warn] 4071352#0: *6523262 [lua] balancer.lua:316: [healthchecks] failed setting peer status (upstream: 5fb68818-d502-41ae-abcc-a41ff1405783:testupstream): no peer found by name '10.3.6.2' and address 10.3.6.2:31072, context: ngx.timer
2021/05/18 11:25:25 [warn] 4071346#0: *6523265 [lua] balancer.lua:316: [healthchecks] failed setting peer status (upstream: 5fb68818-d502-41ae-abcc-a41ff1405783:testupstream): no peer found by name '10.3.6.2' and address 10.3.6.2:31072, context: ngx.timer
2021/05/18 11:25:25 [warn] 4071348#0: *6523267 [lua] balancer.lua:316: [healthchecks] failed setting peer status (upstream: 5fb68818-d502-41ae-abcc-a41ff1405783:testupstream): no peer found by name '10.3.6.2' and address 10.3.6.2:31072, context: ngx.timer
I reported this bug in the issue https://github.com/Kong/kong/issues/6983#issuecomment-811297990
Additional Details & Logs
- Kong 2.4.1
- Cassandra multi DC
- Kong configuration
{
"pids": {
"master": 4067155,
"workers": [
4071344,
4071345,
4071346,
4071347,
4071348,
4071349,
4071350,
4071351,
4071352,
4071353,
4071354,
4071355,
4071356,
4071357,
4071358,
4071359
]
},
"version": "2.4.1",
"hostname": "api-gw-1",
"lua_version": "LuaJIT 2.1.0-beta3",
"node_id": "fbe28f45-0c0c-4f79-acef-322b505c7d46",
"plugins": {
"available_on_server": {
"session": true,
"acme": true,
"grpc-web": true,
"grpc-gateway": true,
"jwt": true,
"acl": true,
"correlation-id": true,
"cors": true,
"oauth2": true,
"tcp-log": true,
"udp-log": true,
"file-log": true,
"http-log": true,
"key-auth": true,
"hmac-auth": true,
"basic-auth": true,
"ip-restriction": true,
"request-transformer": true,
"response-transformer": true,
"request-size-limiting": true,
"rate-limiting": true,
"response-ratelimiting": true,
"syslog": true,
"loggly": true,
"datadog": true,
"ldap-auth": true,
"statsd": true,
"bot-detection": true,
"aws-lambda": true,
"request-termination": true,
"azure-functions": true,
"zipkin": true,
"pre-function": true,
"post-function": true,
"prometheus": true,
"proxy-cache": true
},
"enabled_in_cluster": [
"file-log",
"key-auth",
"prometheus",
"tcp-log",
"rate-limiting",
"cors"
]
},
"configuration": {
"cluster_data_plane_purge_delay": 1209600,
"nginx_err_logs": "/usr/local/kong/logs/error.log",
"pluginserver_names": {},
"admin_acc_logs": "/usr/local/kong/logs/admin_access.log",
"nginx_conf": "/usr/local/kong/nginx.conf",
"nginx_kong_conf": "/usr/local/kong/nginx-kong.conf",
"nginx_kong_stream_conf": "/usr/local/kong/nginx-kong-stream.conf",
"kong_env": "/usr/local/kong/.kong_env",
"client_ssl": false,
"go_plugins_dir": "off",
"ssl_cert_default": "/usr/local/kong/ssl/kong-default.crt",
"prefix": "/usr/local/kong",
"go_pluginserver_exe": "/usr/local/bin/go-pluginserver",
"dns_resolver": {},
"dns_hostsfile": "/etc/hosts",
"dns_error_ttl": 1,
"dns_not_found_ttl": 30,
"dns_stale_ttl": 4,
"dns_order": [
"LAST",
"SRV",
"A",
"CNAME"
],
"lua_package_path": "./?.lua;./?/init.lua;",
"cluster_control_plane": "127.0.0.1:8005",
"cluster_mtls": "shared",
"database": "cassandra",
"ssl_cipher_suite": "intermediate",
"ssl_ciphers": "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384",
"kic": false,
"admin_listeners": [
{
"bind": false,
"backlog=16384": true,
"ip": "0.0.0.0",
"port": 8444,
"ssl": true,
"listener": "0.0.0.0:8444 ssl http2 reuseport backlog=16384",
"proxy_protocol": false,
"deferred": false,
"reuseport": true,
"http2": true
}
],
"proxy_listeners": [
{
"bind": false,
"backlog=16384": true,
"ip": "0.0.0.0",
"port": 8000,
"ssl": false,
"listener": "0.0.0.0:8000 reuseport backlog=16384",
"proxy_protocol": false,
"deferred": false,
"reuseport": true,
"http2": false
},
{
"bind": false,
"backlog=16384": true,
"ip": "0.0.0.0",
"port": 8443,
"ssl": true,
"listener": "0.0.0.0:8443 ssl http2 reuseport backlog=16384",
"proxy_protocol": false,
"deferred": false,
"reuseport": true,
"http2": true
}
],
"stream_listeners": {},
"db_cache_warmup_entities": [
"services"
],
"nginx_http_lua_shared_dict": "prometheus_metrics 10m",
"dns_no_sync": false,
"nginx_http_client_max_body_size": "50m",
"untrusted_lua_sandbox_requires": {},
"untrusted_lua": "sandbox",
"ssl_cert_key_default_ecdsa": "/usr/local/kong/ssl/kong-default-ecdsa.key",
"anonymous_reports": false,
"client_ssl_cert_default": "/usr/local/kong/ssl/kong-default.crt",
"client_ssl_cert_key_default": "/usr/local/kong/ssl/kong-default.key",
"admin_ssl_cert_default": "/usr/local/kong/ssl/admin-kong-default.crt",
"worker_consistency": "strict",
"admin_ssl_cert_key_default": "/usr/local/kong/ssl/admin-kong-default.key",
"pg_semaphore_timeout": 60000,
"admin_ssl_cert_default_ecdsa": "/usr/local/kong/ssl/admin-kong-default-ecdsa.crt",
"lua_socket_pool_size": 30,
"admin_ssl_cert_key_default_ecdsa": "/usr/local/kong/ssl/admin-kong-default-ecdsa.key",
"nginx_proxy_real_ip_header": "X-Real-IP",
"status_ssl_cert_default": "/usr/local/kong/ssl/status-kong-default.crt",
"nginx_http_lua_ssl_protocols": "TLSv1.1 TLSv1.2 TLSv1.3",
"status_ssl_cert_key_default": "/usr/local/kong/ssl/status-kong-default.key",
"upstream_keepalive_max_requests": 1024,
"status_ssl_cert_default_ecdsa": "/usr/local/kong/ssl/status-kong-default-ecdsa.crt",
"cassandra_keyspace": "kong_adt",
"status_ssl_cert_key_default_ecdsa": "/usr/local/kong/ssl/status-kong-default-ecdsa.key",
"lua_package_cpath": "",
"db_cache_ttl": 0,
"nginx_optimizations": true,
"proxy_stream_access_log": "logs/access.log basic",
"upstream_keepalive_pool_size": 60,
"ssl_cert_csr_default": "/usr/local/kong/ssl/kong-default.csr",
"ssl_cert_key_default": "/usr/local/kong/ssl/kong-default.key",
"status_error_log": "logs/status_error.log",
"nginx_acc_logs": "/usr/local/kong/logs/access.log",
"admin_error_log": "logs/error.log",
"status_listen": [
"off"
],
"port_maps": {},
"proxy_listen": [
"0.0.0.0:8000 reuseport backlog=16384",
"0.0.0.0:8443 http2 ssl reuseport backlog=16384"
],
"admin_listen": [
"0.0.0.0:8444 http2 ssl reuseport backlog=16384"
],
"mem_cache_size": "1024m",
"stream_listen": [
"off"
],
"cluster_listen": [
"0.0.0.0:8005"
],
"admin_ssl_cert": [
"/usr/local/kong/ssl/admin-kong-default.crt",
"/usr/local/kong/ssl/admin-kong-default-ecdsa.crt"
],
"admin_ssl_cert_key": [
"/usr/local/kong/ssl/admin-kong-default.key",
"/usr/local/kong/ssl/admin-kong-default-ecdsa.key"
],
"status_ssl_cert": {},
"status_ssl_cert_key": {},
"db_resurrect_ttl": 30,
"nginx_user": "kong kong",
"nginx_main_user": "kong kong",
"nginx_daemon": "on",
"proxy_ssl_enabled": true,
"nginx_main_daemon": "on",
"plugins": [
"bundled"
],
"nginx_main_worker_processes": "auto",
"loaded_plugins": {
"session": true,
"acme": true,
"grpc-web": true,
"grpc-gateway": true,
"jwt": true,
"acl": true,
"correlation-id": true,
"cors": true,
"oauth2": true,
"tcp-log": true,
"udp-log": true,
"file-log": true,
"http-log": true,
"key-auth": true,
"hmac-auth": true,
"basic-auth": true,
"ip-restriction": true,
"request-transformer": true,
"response-transformer": true,
"request-size-limiting": true,
"rate-limiting": true,
"response-ratelimiting": true,
"syslog": true,
"loggly": true,
"datadog": true,
"ldap-auth": true,
"statsd": true,
"bot-detection": true,
"aws-lambda": true,
"request-termination": true,
"azure-functions": true,
"zipkin": true,
"pre-function": true,
"post-function": true,
"prometheus": true,
"proxy-cache": true
},
"trusted_ips": {},
"real_ip_header": "X-Real-IP",
"role": "traditional",
"real_ip_recursive": "off",
"nginx_proxy_real_ip_recursive": "off",
"client_max_body_size": "0",
"error_default_type": "text/plain",
"client_body_buffer_size": "8k",
"nginx_http_client_body_buffer_size": "128k",
"pg_port": 5432,
"nginx_main_worker_rlimit_nofile": "16384",
"pg_ssl": false,
"pg_ssl_verify": false,
"pg_max_concurrent_queries": 0,
"stream_proxy_ssl_enabled": false,
"admin_ssl_enabled": true,
"status_ssl_enabled": false,
"nginx_worker_processes": "auto",
"pg_ro_ssl": false,
"pg_ro_ssl_verify": false,
"nginx_sproxy_directives": {},
"ssl_cert_key": [
"/usr/local/kong/ssl/kong-default.key",
"/usr/local/kong/ssl/kong-default-ecdsa.key"
],
"cassandra_contact_points": [
"10.5.2.2",
"10.5.2.3",
"10.5.2.4",
"10.3.3.99",
"10.3.3.100",
"10.3.3.101"
],
"cassandra_port": 9042,
"cassandra_password": "******",
"cassandra_ssl": false,
"cassandra_ssl_verify": false,
"cassandra_write_consistency": "LOCAL_QUORUM",
"status_listeners": {},
"proxy_access_log": "off",
"nginx_events_multi_accept": "on",
"nginx_http_ssl_session_timeout": "1d",
"nginx_admin_client_max_body_size": "10m",
"cassandra_timeout": 60000,
"pg_timeout": 60000,
"nginx_admin_client_body_buffer_size": "10m",
"cassandra_read_consistency": "LOCAL_QUORUM",
"worker_state_update_frequency": 5,
"ssl_cert_default_ecdsa": "/usr/local/kong/ssl/kong-default-ecdsa.crt",
"cassandra_lb_policy": "RequestDCAwareRoundRobin",
"nginx_http_ssl_session_tickets": "on",
"db_update_frequency": 5,
"cassandra_repl_strategy": "NetworkTopologyStrategy",
"upstream_keepalive_idle_timeout": 60,
"cassandra_local_datacenter": "ntl",
"cassandra_refresh_frequency": 60,
"headers": [
"off"
],
"cassandra_repl_factor": 3,
"pg_host": "127.0.0.1",
"enabled_headers": {
"Via": false,
"Server": false,
"X-Kong-Proxy-Latency": false,
"X-Kong-Response-Latency": false,
"X-Kong-Admin-Latency": false,
"X-Kong-Upstream-Latency": false,
"X-Kong-Upstream-Status": false,
"server_tokens": false,
"latency_tokens": false
},
"cassandra_data_centers": [
"nps:3",
"ntl:3"
],
"cassandra_schema_consensus_timeout": 60000,
"pg_database": "kong",
"nginx_stream_ssl_prefer_server_ciphers": "off",
"lua_ssl_verify_depth": 1,
"ssl_protocols": "TLSv1.1 TLSv1.2 TLSv1.3",
"nginx_stream_ssl_protocols": "TLSv1.2 TLSv1.3",
"nginx_http_ssl_protocols": "TLSv1.2 TLSv1.3",
"pg_user": "kong",
"ssl_prefer_server_ciphers": "on",
"nginx_http_ssl_prefer_server_ciphers": "off",
"ssl_cert": [
"/usr/local/kong/ssl/kong-default.crt",
"/usr/local/kong/ssl/kong-default-ecdsa.crt"
],
"ssl_dhparam": "ffdhe2048",
"nginx_http_ssl_dhparam": "ffdhe2048",
"nginx_stream_ssl_dhparam": "ffdhe2048",
"ssl_session_tickets": "on",
"nginx_main_directives": [
{
"name": "daemon",
"value": "on"
},
{
"name": "user",
"value": "kong kong"
},
{
"name": "worker_processes",
"value": "auto"
},
{
"name": "worker_rlimit_nofile",
"value": "16384"
}
],
"nginx_stream_ssl_session_tickets": "on",
"ssl_session_timeout": "1d",
"nginx_events_directives": [
{
"name": "multi_accept",
"value": "on"
},
{
"name": "worker_connections",
"value": "16384"
}
],
"nginx_stream_ssl_session_timeout": "1d",
"nginx_http_directives": [
{
"name": "client_body_buffer_size",
"value": "128k"
},
{
"name": "client_max_body_size",
"value": "50m"
},
{
"name": "lua_shared_dict",
"value": "prometheus_metrics 10m"
},
{
"name": "lua_ssl_protocols",
"value": "TLSv1.1 TLSv1.2 TLSv1.3"
},
{
"name": "ssl_dhparam",
"value": "/usr/local/kong/ssl/ffdhe2048.pem"
},
{
"name": "ssl_prefer_server_ciphers",
"value": "off"
},
{
"name": "ssl_protocols",
"value": "TLSv1.2 TLSv1.3"
},
{
"name": "ssl_session_tickets",
"value": "on"
},
{
"name": "ssl_session_timeout",
"value": "1d"
}
],
"proxy_error_log": "logs/error.log",
"cluster_ocsp": "off",
"proxy_stream_error_log": "logs/error.log",
"admin_access_log": "logs/admin_access.log",
"nginx_upstream_directives": {},
"status_access_log": "off",
"nginx_proxy_directives": [
{
"name": "real_ip_header",
"value": "X-Real-IP"
},
{
"name": "real_ip_recursive",
"value": "off"
}
],
"log_level": "notice",
"nginx_status_directives": {},
"db_update_propagation": 2,
"nginx_admin_directives": [
{
"name": "client_body_buffer_size",
"value": "10m"
},
{
"name": "client_max_body_size",
"value": "10m"
}
],
"lua_ssl_trusted_certificate": {},
"nginx_stream_directives": [
{
"name": "lua_shared_dict",
"value": "stream_prometheus_metrics 5m"
},
{
"name": "lua_ssl_protocols",
"value": "TLSv1.1 TLSv1.2 TLSv1.3"
},
{
"name": "ssl_dhparam",
"value": "/usr/local/kong/ssl/ffdhe2048.pem"
},
{
"name": "ssl_prefer_server_ciphers",
"value": "off"
},
{
"name": "ssl_protocols",
"value": "TLSv1.2 TLSv1.3"
},
{
"name": "ssl_session_tickets",
"value": "on"
},
{
"name": "ssl_session_timeout",
"value": "1d"
}
],
"lua_ssl_protocols": "TLSv1.1 TLSv1.2 TLSv1.3",
"nginx_supstream_directives": {},
"nginx_stream_lua_ssl_protocols": "TLSv1.1 TLSv1.2 TLSv1.3",
"cluster_listeners": [
{
"ssl": false,
"ip": "0.0.0.0",
"listener": "0.0.0.0:8005",
"port": 8005,
"bind": false,
"http2": false,
"proxy_protocol": false,
"deferred": false,
"reuseport": false,
"backlog=%d+": false
}
],
"cassandra_username": "kong_adt",
"host_ports": {},
"nginx_events_worker_connections": "16384",
"nginx_http_upstream_directives": {},
"nginx_http_status_directives": {},
"untrusted_lua_sandbox_environment": {},
"nginx_pid": "/usr/local/kong/pids/nginx.pid"
},
"tagline": "Welcome to kong",
"timers": {
"running": 212,
"pending": 7
}
}
- Ubuntu 18.04
- Upstream info
{
"hash_on_header": null,
"created_at": 1621310517,
"algorithm": "round-robin",
"host_header": null,
"id": "349d3f5b-4bcc-48ac-a741-3d60d9bfb118",
"hash_on_cookie": null,
"hash_on_cookie_path": "/",
"hash_fallback": "none",
"hash_fallback_header": null,
"name": "testupstream",
"tags": null,
"slots": 10000,
"client_certificate": null,
"hash_on": "none",
"healthchecks": {
"passive": {
"unhealthy": {
"http_failures": 0,
"tcp_failures": 0,
"http_statuses": [
429,
500,
503
],
"timeouts": 0
},
"type": "http",
"healthy": {
"http_statuses": [
200,
201,
202,
203,
204,
205,
206,
207,
208,
226,
300,
301,
302,
303,
304,
305,
306,
307,
308
],
"successes": 0
}
},
"threshold": 0,
"active": {
"concurrency": 1,
"type": "http",
"http_path": "/ping",
"https_sni": null,
"https_verify_certificate": false,
"timeout": 5,
"unhealthy": {
"http_statuses": [
429,
404,
500,
501,
502,
503,
504,
505
],
"interval": 10,
"http_failures": 5,
"tcp_failures": 5,
"timeouts": 5
},
"healthy": {
"successes": 1,
"http_statuses": [
200,
302
],
"interval": 10
}
}
}
}
In my Kong config file, I'm setting db_update_frequency = 5, db_update_propagation = 2
When I deleted a target from an Upstream but backend service still receive health check requests from Kong
I waited ~10m then ran kong reload but issue still existed until I ran kong restart
In above case, I updated health check path from /ping to / then check backend service, I got 2 request come from Kong (one to / and one to /ping)
10.5.93.173 - - [18/May/2021 22:25:28] "GET /ping HTTP/1.0" 404 -
10.5.93.173 - - [18/May/2021 22:25:29] "GET / HTTP/1.0" 200 -
10.5.93.173 - - [18/May/2021 22:25:29] "GET / HTTP/1.0" 200 -
10.5.93.173 - - [18/May/2021 22:25:33] "GET /ping HTTP/1.0" 404 -
Upstream health status has changed from UNHEALTHY to HEALTHY and vice versa continuously
- Kong debug logs:
2021/05/18 22:31:00 [debug] 2436736#0: *264896 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) checking unhealthy targets: #1
2021/05/18 22:31:00 [debug] 2436736#0: *264896 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) Checking 10.2.3.2 10.2.3.2:8999 (currently unhealthy)
2021/05/18 22:31:00 [debug] 2436736#0: *264896 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) Reporting '10.2.3.2 (10.2.3.2:8999)' (got HTTP 404)
2021/05/18 22:31:00 [debug] 2436735#0: *264899 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh], event=unhealthy, pid=2436736, data=table: 0x7fbe824a4660
2021/05/18 22:31:00 [debug] 2436735#0: *264899 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) event: target status '10.2.3.2(10.2.3.2:8999)' from 'true' to 'false'
2021/05/18 22:31:00 [debug] 2436735#0: *264899 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) event: target status '10.2.3.2(10.2.3.2:8999)' from 'true' to 'false'
2021/05/18 22:31:00 [debug] 2436735#0: *264899 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) event: target status '10.2.3.2(10.2.3.2:8999)' from 'true' to 'false'
2021/05/18 22:31:00 [warn] 2436735#0: *264899 [lua] balancer.lua:316: [healthchecks] failed setting peer status (upstream: 2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh): no peer found by name '10.2.3.2' and address 10.2.3.2:8999, context: ngx.timer
2021/05/18 22:31:00 [debug] 2436735#0: *264899 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) event: target status '10.2.3.2(10.2.3.2:8999)' from 'true' to 'false'
2021/05/18 22:31:00 [debug] 2436735#0: *264899 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) event: target status '10.2.3.2(10.2.3.2:8999)' from 'true' to 'false'
2021/05/18 22:31:00 [warn] 2436735#0: *264900 [lua] balancer.lua:258: callback(): [healthchecks] balancer 2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh reported health status changed to UNHEALTHY, context: ngx.timer
2021/05/18 22:31:00 [debug] 2436737#0: *264902 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh], event=unhealthy, pid=2436736, data=table: 0x7fbe81c27870
2021/05/18 22:31:00 [debug] 2436737#0: *264902 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) event: target status '10.2.3.2(10.2.3.2:8999)' from 'true' to 'false'
2021/05/18 22:31:00 [debug] 2436737#0: *264902 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) event: target status '10.2.3.2(10.2.3.2:8999)' from 'true' to 'false'
2021/05/18 22:31:00 [debug] 2436737#0: *264902 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) event: target status '10.2.3.2(10.2.3.2:8999)' from 'true' to 'false'
2021/05/18 22:31:00 [debug] 2436737#0: *264902 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) event: target status '10.2.3.2(10.2.3.2:8999)' from 'true' to 'false'
2021/05/18 22:31:00 [debug] 2436737#0: *264902 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) event: target status '10.2.3.2(10.2.3.2:8999)' from 'true' to 'false'
2021/05/18 22:31:00 [warn] 2436737#0: *264904 [lua] balancer.lua:258: callback(): [healthchecks] balancer 2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh reported health status changed to UNHEALTHY, context: ngx.timer
2021/05/18 22:31:01 [debug] 2436737#0: *264911 [lua] init.lua:288: [cluster_events] polling events from: 1621351305.981
2021/05/18 22:31:01 [debug] 2436737#0: *264911 [lua] cluster.lua:481: next_coordinator(): [lua-cassandra] load balancing policy chose host at 10.5.2.2
2021/05/18 22:31:01 [debug] 2436736#0: *264915 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) checking unhealthy targets: #1
2021/05/18 22:31:01 [debug] 2436736#0: *264915 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) Checking 10.2.3.2 10.2.3.2:8999 (currently unhealthy)
2021/05/18 22:31:01 [debug] 2436736#0: *264915 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) Reporting '10.2.3.2 (10.2.3.2:8999)' (got HTTP 200)
2021/05/18 22:31:01 [warn] 2436736#0: *264915 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) healthy SUCCESS increment (1/1) for '10.2.3.2(10.2.3.2:8999)', context: ngx.timer
2021/05/18 22:31:01 [debug] 2436736#0: *264915 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh], event=healthy, pid=2436736, data=table: 0x7fbe82bb9b00
2021/05/18 22:31:01 [debug] 2436736#0: *264915 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) event: target status '10.2.3.2(10.2.3.2:8999)' from 'false' to 'true'
2021/05/18 22:31:01 [debug] 2436736#0: *264915 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) event: target status '10.2.3.2(10.2.3.2:8999)' from 'false' to 'true'
2021/05/18 22:31:01 [debug] 2436736#0: *264915 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) event: target status '10.2.3.2(10.2.3.2:8999)' from 'false' to 'true'
2021/05/18 22:31:01 [debug] 2436736#0: *264915 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) event: target status '10.2.3.2(10.2.3.2:8999)' from 'false' to 'true'
2021/05/18 22:31:01 [warn] 2436736#0: *264917 [lua] balancer.lua:258: callback(): [healthchecks] balancer 2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh reported health status changed to HEALTHY, context: ngx.timer
2021/05/18 22:31:02 [debug] 2436736#0: *264918 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) checking healthy targets: #1
2021/05/18 22:31:02 [debug] 2436736#0: *264918 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) Checking 10.2.3.2 10.2.3.2:8999 (currently healthy)
2021/05/18 22:31:02 [debug] 2436736#0: *264918 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) Reporting '10.2.3.2 (10.2.3.2:8999)' (got HTTP 200)
2021/05/18 22:31:02 [debug] 2436738#0: *264924 [lua] events.lua:211: do_event_json(): worker-events: handling event; source=lua-resty-healthcheck [2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh], event=healthy, pid=2436736, data=table: 0x7fbe82eff420
2021/05/18 22:31:02 [debug] 2436738#0: *264924 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) event: target status '10.2.3.2(10.2.3.2:8999)' from 'false' to 'true'
2021/05/18 22:31:02 [debug] 2436738#0: *264924 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) event: target status '10.2.3.2(10.2.3.2:8999)' from 'false' to 'true'
2021/05/18 22:31:02 [debug] 2436738#0: *264924 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) event: target status '10.2.3.2(10.2.3.2:8999)' from 'false' to 'true'
2021/05/18 22:31:02 [warn] 2436738#0: *264924 [lua] balancer.lua:316: [healthchecks] failed setting peer status (upstream: 2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh): no peer found by name '10.2.3.2' and address 10.2.3.2:8999, context: ngx.timer
2021/05/18 22:31:02 [debug] 2436738#0: *264924 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) event: target status '10.2.3.2(10.2.3.2:8999)' from 'false' to 'true'
2021/05/18 22:31:02 [debug] 2436738#0: *264924 [lua] healthcheck.lua:1124: log(): [healthcheck] (2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh) event: target status '10.2.3.2(10.2.3.2:8999)' from 'false' to 'true'
2021/05/18 22:31:02 [warn] 2436738#0: *264925 [lua] balancer.lua:258: callback(): [healthchecks] balancer 2c50842b-3a81-49e9-a96c-8af19d0ac7c6:khanh reported health status changed to HEALTHY, context: ngx.timer
Thanks for you report, @mrpk1906! It seems Kong is not updating correctly the "http_path" and adding a different health check to the same target in your case, that's why it keeps flipping the health status. That behavior might be a bug, we will investigate it.
Hi @locao,
About my case, sometimes Kong didn't execute health check to a target or do request (got error from response status) but still detect it is HEALTHY, any suggest to debug this case? or can you try reproduce my issue?
Then the health check completely failed
i also have the same issue reported by mrpk1906, where target shows healthy but actually target is dead on upstream, seems Kong is not even checking the health-check, however when i restart the kong it starts again checking the health-check as expected and mark it unhealthy. seems some cache issue, not sure though, did anyone found the solution for this?.
I haven't tried to reproduce this lately. Sounds like a bug we fixed in the 2.x series, but I am not sure. If we can get the reproduction steps for a recent version, I can give a shot on fixing.
This issue is marked as stale because it has been open for 14 days with no activity.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
@chronolaw update lib/resty/healthcheck.lua file as mention in pull request. https://github.com/Kong/lua-resty-healthcheck/pull/78/files
This issue is marked as stale because it has been open for 14 days with no activity.
Dear contributor,
We are automatically closing this issue because it has not seen any activity for three weeks. We're sorry that your issue could not be resolved. If any new information comes up that could help resolving it, please feel free to reopen it.
Your contribution is greatly appreciated!
Please have a look our pledge to the community for more information.
Sincerely, Your Kong Gateway team