tengine 瞬间502问题

[swq499809608]

系统环境及版本： OS: CentOS release 6.9 (Final)

Tengine 版本：/usr/local/nginx/sbin/nginx -tv Tengine version: Tengine/2.3.0 (nginx/1.15.9) nginx: the configuration file /usr/local/nginx/conf/nginx.conf syntax is ok

nginx 实例配置： upstream 7461089c0c13e852e951bc9c96caf781 { server xxx.xxx.xx.xxx:8888 weight=150 max_fails=2 fail_timeout=6s; server xxx.xxx.xx.xxx:9999 weight=150 max_fails=2 fail_timeout=6s; } server { server_name 520ts.xx.com.cn; listen 80; index index.aspx index.shtml index.html; access_log /data/nginx/logs/520ts.xx.com.cn_access.log access; error_log /data/nginx/logs/520ts.xx.com.cn_access.debug debug; keepalive_timeout 65; location ~ /.git { deny all; } location / { proxy_pass http://7461089c0c13e852e951bc9c96caf781; } }

问题： 当upstream 某一台主机出现响应慢or超时会出现后端不响应，瞬间出现 504->502 的现象，在fail_timeout = 6秒内会瞬间502 的异常 通过版本测试，得出结论： Openresty，Tengine 的Nginx 版本<= 1.11 版本不会出现瞬间502的情况 版本>=1.12 版本会出现瞬间502的情况

实验环境： upstream 站点启动 8888、9999 server，通过Nginx 代理，提供post接口和get 接口，主动发起502 连续4次即可复现

异常日志： 2022-01-11T14:31:56+08:00 /health 502 152 - 0.000 0.000 520ts.xx.com.cn xxx.xxx.xxx.249 xxx.xxx.214.232 7461089c0c13e852e951bc9c96caf781 - curl/7.29.502 http HEAD @- 2022-01-11T14:31:57+08:00 /health 502 152 - 0.000 0.000 520ts.xx.com.cn xxx.xxx.xxx.249 xxx.xxx.214.232 7461089c0c13e852e951bc9c96caf781 - curl/7.29.502 http HEAD @- 2022-01-11T14:31:57+08:00 /health 502 152 - 0.000 0.000 520ts.xx.com.cn xxx.xxx.xxx.249 xxx.xxx.214.232 7461089c0c13e852e951bc9c96caf781 - curl/7.29.502 http HEAD @- 2022-01-11T14:31:58+08:00 /health 502 152 - 0.000 0.000 520ts.xx.com.cn xxx.xxx.xxx.249 xxx.xxx.214.232 7461089c0c13e852e951bc9c96caf781 - curl/7.29.502 http HEAD @- 2022-01-11T14:31:58+08:00 /health 502 152 - 0.000 0.000 520ts.xx.com.cn xxx.xxx.xxx.249 xxx.xxx.214.232 7461089c0c13e852e951bc9c96caf781 - curl/7.29.502 http HEAD @- 2022-01-11T14:31:58+08:00 /health 502 152 - 0.000 0.000 520ts.xx.com.cn xxx.xxx.xxx.249 xxx.xxx.214.232 7461089c0c13e852e951bc9c96caf781 - curl/7.29.502 http HEAD @- 2022-01-11T14:31:58+08:00 /health 502 152 - 0.000 0.000 520ts.xx.com.cn xxx.xxx.xxx.249 xxx.xxx.214.232 7461089c0c13e852e951bc9c96caf781 - curl/7.29.502 http HEAD @-

Debug 日志：

2022/01/11 14:31:58 [error] 108996#0: *24822 no live upstreams while connecting to upstream, client: xxx.xxx.220.249, server: 520ts.xx.com.cn, request: "HEAD /health HTTP/1.1", upstream: "http://7461089c0c13e852e951bc9c96caf781/health", host: "520ts.xx.com.cn" 2022/01/11 14:31:59 [error] 108996#0: *24853 no live upstreams while connecting to upstream, client: xxx.xxx.220.249, server: 520ts.xx.com.cn, request: "HEAD /health HTTP/1.1", upstream: "http://7461089c0c13e852e951bc9c96caf781/health", host: "520ts.xx.com.cn" 2022/01/11 14:31:59 [error] 108996#0: *24862 no live upstreams while connecting to upstream, client: xxx.xxx.220.249, server: 520ts.xx.com.cn, request: "HEAD /health HTTP/1.1", upstream: "http://7461089c0c13e852e951bc9c96caf781/health", host: "520ts.xx.com.cn" 2022/01/11 14:31:59 [error] 108996#0: *24870 no live upstreams while connecting to upstream, client: xxx.xxx.220.249, server: 520ts.xx.com.cn, request: "HEAD /health HTTP/1.1", upstream: "http://7461089c0c13e852e951bc9c96caf781/health", host: "520ts.xx.com.cn" 2022/01/11 14:31:59 [error] 108996#0: *24924 no live upstreams while connecting to upstream, client: xxx.xxx.220.249, server: 520ts.xx.com.cn, request: "HEAD /health HTTP/1.1", upstream: "http://7461089c0c13e852e951bc9c96caf781/health", host: "520ts.xx.com.cn" 2022/01/11 14:31:59 [error] 108996#0: *24929 no live upstreams while connecting to upstream, client: xxx.xxx.220.249, server: 520ts.xx.com.cn, request: "HEAD /health HTTP/1.1", upstream: "http://7461089c0c13e852e951bc9c96caf781/health", host: "520ts.xx.com.cn" 2022/01/11 14:32:01 [error] 108996#0: *25124 no live upstreams while connecting to upstream, client: xxx.xxx.220.249, server: 520ts.xx.com.cn, request: "HEAD /health HTTP/1.1", upstream: "http://7461089c0c13e852e951bc9c96caf781/health", host: "520ts.xx.com.cn" 2022/01/11 14:32:01 [error] 108996#0: *25135 no live upstreams while connecting to upstream, client: xxx.xxx.220.249, server: 520ts.xx.com.cn, request: "HEAD /health HTTP/1.1", upstream: "http://7461089c0c13e852e951bc9c96caf781/health", host: "520ts.xx.com.cn" 2022/01/11 14:32:01 [error] 108996#0: *25208 no live upstreams while connecting to upstream, client: xxx.xxx.220.249, server: 520ts.xx.com.cn, request: "HEAD /health HTTP/1.1", upstream: "http://7461089c0c13e852e951bc9c96caf781/health", host: "520ts.xx.com.cn"

目前解决的办法： 1、关闭被动检查(max_fails = 0 ) 2、配置主动检测tcp或http（nginx_upstream_check_module)