kubernetes-ingress
kubernetes-ingress copied to clipboard
Published
20 hours ago •
nginxinc
nginxinc
Receive Upstream Timeout from IP which not present on k8s
Describe the bug Hi, we are using VirtualServer CRD for configure route on k8s to send traffic to the upstream backend After restart upstream server (backend) which has temproary IP address, our nginx ingress continue send traffic to the IP address which is not present yet.
To Reproduce Steps to reproduce the behavior:
- Deploy x to '...' using some.yaml
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
name: pn-front-prod-arbitrum-sepolia-rpc
namespace: public
spec:
server-snippets: |
proxy_request_buffering off;
ssl_buffer_size 4k;
host: arbitrum-sepolia-rpc.example.com
tls:
secret: example.com
upstreams:
- name: backend
service: pn-backend
port: 4000
- name: frontend
service: pn-frontend
port: 3000
routes:
- path: /api/metrics
matches:
- conditions:
- variable: $request_method
value: GET
action:
redirect:
url: https://arbitrum-sepolia-rpc.example.com/
code: 301
action:
pass: frontend
- path: /api
matches:
- conditions:
- variable: $request_method
value: GET
action:
pass: frontend
action:
pass: frontend
- path: /favicon.ico
matches:
- conditions:
- variable: $request_method
value: GET
action:
pass: frontend
action:
pass: frontend
- path: /platforms
matches:
- conditions:
- variable: $request_method
value: GET
action:
pass: frontend
action:
pass: frontend
- path: /_next
matches:
- conditions:
- variable: $request_method
value: GET
action:
pass: frontend
action:
pass: frontend
- path: /
matches:
- conditions:
- header: Upgrade
value: websocket
action:
pass: backend
- conditions:
- variable: $request_method
value: GET
action:
proxy:
upstream: frontend
rewritePath: /arbitrum-sepolia
action:
pass: backend
- View logs on '....'
date="2024-04-04T14:03:01+00:00" status=200 request_completion=OK msec=1712239381.780 connections_active=48 connections_reading=0 connections_writing=138 connections_waiting=24 connection=21143130 connection_requests=80 connection_time=1423.508 client=10.244.8.117 method=POST request="POST /?testbot&time=1712239321732 HTTP/2.0" request_length=240 status=200 bytes_sent=205 body_bytes_sent=46 referer= user_agent="Go-http-client/2.0" upstream_addr=10.244.21.150:4000, 10.244.24.58:4000 upstream_status=504, 200 request_time=60.008 upstream_response_time=60.002, 0.006 upstream_connect_time=-, 0.000 upstream_header_time=-, 0.006 request_body="{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"eth_blockNumber\",\"params\":[]} host="[arbitrum-sepolia-rpc.example.com](http://arbitrum-sepolia-rpc.example.com/)" user_ip="164.90.160.159"
- See error
upstream_status=504, 200
Expected behavior
[nariman@notebook new-only-back]$ kubectl -n public get endpoints
NAME ENDPOINTS AGE
pn-backend 10.244.0.11:4001,10.244.0.163:4001,10.244.11.130:4001 + 92 more... 387d
If uptream IP address not present into endpoints why nginx try to send traffic to non-existent IP Your environment
- Version of the Ingress Controller - release version or a specific commit
[nariman@notebook new-only-back]$ helm ls -n nginx
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
ingress nginx 19 2024-03-29 12:17:00.734183664 +0400 +04 deployed nginx-ingress-1.1.0 3.4.0
- Version of Kubernetes
[nariman@notebook new-only-back]$ kubectl version
Client Version: v1.29.2
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.27.10
- Kubernetes platform (e.g. Mini-kube or GCP)
DigitalOcean - Using NGINX or NGINX Plus
NGINX
Additional context Add any other context about the problem here. Any log files you want to share. Config inside ingress controller
nginx@ingress-nginx-ingress-controller-565c6849d5-4k6kf:/etc/nginx/conf.d$ cat vs_public_pn-front-prod-arbitrum-sepolia-rpc.conf
upstream vs_public_pn-front-prod-arbitrum-sepolia-rpc_backend {zone vs_public_pn-front-prod-arbitrum-sepolia-rpc_backend 256k;random two least_conn;
server 10.244.0.11:4000 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.0.163:4000 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.11.130:4000 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.2.141:4000 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.24.217:4000 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.24.99:4000 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.3.157:4000 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.37.252:4000 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.39.214:4000 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.4.89:4000 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.40.54:4000 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.42.195:4000 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.43.186:4000 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.47.216:4000 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.48.184:4000 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.49.232:4000 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.51.146:4000 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.51.85:4000 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.52.54:4000 max_fails=1 fail_timeout=10s max_conns=0;
}
upstream vs_public_pn-front-prod-arbitrum-sepolia-rpc_frontend {zone vs_public_pn-front-prod-arbitrum-sepolia-rpc_frontend 256k;random two least_conn;
server 10.244.21.9:3000 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.3.22:3000 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.36.163:3000 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.38.151:3000 max_fails=1 fail_timeout=10s max_conns=0;
server 10.244.5.10:3000 max_fails=1 fail_timeout=10s max_conns=0;
}
map $request_method $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_0_match_0_cond_0 {
"GET" 1;
default 0;
}
map $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_0_match_0_cond_0 $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_0 {
~^1 /internal_location_matches_0_match_0;
default /internal_location_matches_0_default;
}
map $request_method $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_1_match_0_cond_0 {
"GET" 1;
default 0;
}
map $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_1_match_0_cond_0 $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_1 {
~^1 /internal_location_matches_1_match_0;
default /internal_location_matches_1_default;
}
map $request_method $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_2_match_0_cond_0 {
"GET" 1;
default 0;
}
map $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_2_match_0_cond_0 $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_2 {
~^1 /internal_location_matches_2_match_0;
default /internal_location_matches_2_default;
}
map $request_method $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_3_match_0_cond_0 {
"GET" 1;
default 0;
}
map $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_3_match_0_cond_0 $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_3 {
~^1 /internal_location_matches_3_match_0;
default /internal_location_matches_3_default;
}
map $request_method $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_4_match_0_cond_0 {
"GET" 1;
default 0;
}
map $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_4_match_0_cond_0 $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_4 {
~^1 /internal_location_matches_4_match_0;
default /internal_location_matches_4_default;
}
map $http_Upgrade $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_5_match_0_cond_0 {
"websocket" 1;
default 0;
}
map $request_method $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_5_match_1_cond_0 {
"GET" 1;
default 0;
}
map $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_5_match_0_cond_0$vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_5_match_1_cond_0 $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_5 {
~^1 /internal_location_matches_5_match_0;
~^01 /internal_location_matches_5_match_1;
default /internal_location_matches_5_default;
}
server {
listen 80;
listen [::]:80;
server_name arbitrum-sepolia-rpc.example.com;
set $resource_type "virtualserver";
set $resource_name "pn-front-prod-arbitrum-sepolia-rpc";
set $resource_namespace "public";
listen 443 ssl;
listen [::]:443 ssl;
http2 on;
ssl_certificate $secret_dir_path/public-example.com;
ssl_certificate_key $secret_dir_path/public-example.com;
server_tokens "on";
location /api/metrics {
rewrite ^ $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_0 last;
}
location /api {
rewrite ^ $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_1 last;
}
location /favicon.ico {
rewrite ^ $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_2 last;
}
location /platforms {
rewrite ^ $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_3 last;
}
location /_next {
rewrite ^ $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_4 last;
}
location / {
rewrite ^ $vs_public_pn_front_prod_arbitrum_sepolia_rpc_matches_5 last;
}
location /internal_location_matches_0_match_0 {
set $service "";
error_page 418 =301 "https://arbitrum-sepolia-rpc.example.com/";
proxy_intercept_errors on;
proxy_pass http://unix:/var/lib/nginx/nginx-418-server.sock;
}
location /internal_location_matches_0_default {
set $service "pn-frontend";
internal;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 100m;
proxy_buffering off;
proxy_http_version 1.1;
set $default_connection_header close;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_pass_request_headers on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host "$host";
proxy_pass http://vs_public_pn-front-prod-arbitrum-sepolia-rpc_frontend$request_uri;
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0s;
proxy_next_upstream_tries 0;
}
location /internal_location_matches_1_match_0 {
set $service "pn-frontend";
internal;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 100m;
proxy_buffering off;
proxy_http_version 1.1;
set $default_connection_header close;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_pass_request_headers on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host "$host";
proxy_pass http://vs_public_pn-front-prod-arbitrum-sepolia-rpc_frontend$request_uri;
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0s;
proxy_next_upstream_tries 0;
}
location /internal_location_matches_1_default {
set $service "pn-frontend";
internal;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 100m;
proxy_buffering off;
proxy_http_version 1.1;
set $default_connection_header close;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_pass_request_headers on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host "$host";
proxy_pass http://vs_public_pn-front-prod-arbitrum-sepolia-rpc_frontend$request_uri;
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0s;
proxy_next_upstream_tries 0;
}
location /internal_location_matches_2_match_0 {
set $service "pn-frontend";
internal;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 100m;
proxy_buffering off;
proxy_http_version 1.1;
set $default_connection_header close;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_pass_request_headers on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host "$host";
proxy_pass http://vs_public_pn-front-prod-arbitrum-sepolia-rpc_frontend$request_uri;
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0s;
proxy_next_upstream_tries 0;
}
location /internal_location_matches_2_default {
set $service "pn-frontend";
internal;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 100m;
proxy_buffering off;
proxy_http_version 1.1;
set $default_connection_header close;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_pass_request_headers on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host "$host";
proxy_pass http://vs_public_pn-front-prod-arbitrum-sepolia-rpc_frontend$request_uri;
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0s;
proxy_next_upstream_tries 0;
}
location /internal_location_matches_3_match_0 {
set $service "pn-frontend";
internal;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 100m;
proxy_buffering off;
proxy_http_version 1.1;
set $default_connection_header close;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_pass_request_headers on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host "$host";
proxy_pass http://vs_public_pn-front-prod-arbitrum-sepolia-rpc_frontend$request_uri;
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0s;
proxy_next_upstream_tries 0;
}
location /internal_location_matches_3_default {
set $service "pn-frontend";
internal;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 100m;
proxy_buffering off;
proxy_http_version 1.1;
set $default_connection_header close;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_pass_request_headers on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host "$host";
proxy_pass http://vs_public_pn-front-prod-arbitrum-sepolia-rpc_frontend$request_uri;
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0s;
proxy_next_upstream_tries 0;
}
location /internal_location_matches_4_match_0 {
set $service "pn-frontend";
internal;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 100m;
proxy_buffering off;
proxy_http_version 1.1;
set $default_connection_header close;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_pass_request_headers on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host "$host";
proxy_pass http://vs_public_pn-front-prod-arbitrum-sepolia-rpc_frontend$request_uri;
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0s;
proxy_next_upstream_tries 0;
}
location /internal_location_matches_4_default {
set $service "pn-frontend";
internal;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 100m;
proxy_buffering off;
proxy_http_version 1.1;
set $default_connection_header close;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_pass_request_headers on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host "$host";
proxy_pass http://vs_public_pn-front-prod-arbitrum-sepolia-rpc_frontend$request_uri;
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0s;
proxy_next_upstream_tries 0;
}
location /internal_location_matches_5_match_0 {
set $service "pn-backend";
internal;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 100m;
proxy_buffering off;
proxy_http_version 1.1;
set $default_connection_header close;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_pass_request_headers on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host "$host";
proxy_pass http://vs_public_pn-front-prod-arbitrum-sepolia-rpc_backend$request_uri;
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0s;
proxy_next_upstream_tries 0;
}
location /internal_location_matches_5_match_1 {
set $service "pn-frontend";
internal;
rewrite ^ $request_uri_no_args;
rewrite "^/(.*)$" "/arbitrum-sepolia$1" break;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 100m;
proxy_buffering off;
proxy_http_version 1.1;
set $default_connection_header close;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_pass_request_headers on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host "$host";
proxy_pass http://vs_public_pn-front-prod-arbitrum-sepolia-rpc_frontend;
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0s;
proxy_next_upstream_tries 0;
}
location /internal_location_matches_5_default {
set $service "pn-backend";
internal;
proxy_connect_timeout 60s;
proxy_read_timeout 60s;
proxy_send_timeout 60s;
client_max_body_size 100m;
proxy_buffering off;
proxy_http_version 1.1;
set $default_connection_header close;
proxy_set_header Upgrade $http_upgrade;
proxy_set_header Connection $vs_connection_header;
proxy_pass_request_headers on;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host "$host";
proxy_pass http://vs_public_pn-front-prod-arbitrum-sepolia-rpc_backend$request_uri;
proxy_next_upstream error timeout;
proxy_next_upstream_timeout 0s;
proxy_next_upstream_tries 0;
}
location @grpc_deadline_exceeded {
default_type application/grpc;
add_header content-type application/grpc;
add_header grpc-status 4;
add_header grpc-message 'deadline exceeded';
return 204;
}
location @grpc_permission_denied {
default_type application/grpc;
add_header content-type application/grpc;
add_header grpc-status 7;
add_header grpc-message 'permission denied';
return 204;
}
location @grpc_resource_exhausted {
default_type application/grpc;
add_header content-type application/grpc;
add_header grpc-status 8;
add_header grpc-message 'resource exhausted';
return 204;
}
location @grpc_unimplemented {
default_type application/grpc;
add_header content-type application/grpc;
add_header grpc-status 12;
add_header grpc-message unimplemented;
return 204;
}
location @grpc_internal {
default_type application/grpc;
add_header content-type application/grpc;
add_header grpc-status 13;
add_header grpc-message 'internal error';
return 204;
}
location @grpc_unavailable {
default_type application/grpc;
add_header content-type application/grpc;
add_header grpc-status 14;
add_header grpc-message unavailable;
return 204;
}
location @grpc_unauthenticated {
default_type application/grpc;
add_header content-type application/grpc;
add_header grpc-status 16;
add_header grpc-message unauthenticated;
return 204;
}
}
Hi @ibadullaev-inc4 thanks for reporting!
Be sure to check out the docs and the Contributing Guidelines while you wait for a human to take a look at this :slightly_smiling_face:
Cheers!
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
After restart upstream server (backend) which has temproary IP address, our nginx ingress continue send traffic to the IP address which is not present yet.
NGINX Ingress Controller configures upstreams using endpointSlices and only those endpoints that also are 'ready'. The exception to this would be externalName services, these rely on DNS resolution and the NGINX resolver.
Can you help me understand your scenario a bit deeper? Are these back-end K8s services? ExternalName services?
If it is a timing issue we recommend using a healthcheck https://docs.nginx.com/nginx-ingress-controller/configuration/virtualserver-and-virtualserverroute-resources/#upstreamhealthcheck
Hi, thank you for your response Upstream server is k8s pods ExternalName services? No we do not use ExternalName services
[nariman@notebook new-only-back]$ kubectl -n public get svc pn-backend -o yaml
apiVersion: v1
kind: Service
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"v1","kind":"Service","metadata":{"annotations":{},"name":"pn-backend","namespace":"public"},"spec":{"ports":[{"name":"http","port":4000,"protocol":"TCP","targetPort":4000},{"name":"grpc","port":9090,"protocol":"TCP","targetPort":9090}],"selector":{"app":"pn-backend"},"type":"NodePort"}}
kubernetes.io/change-cause: kubectl edit svc pn-backend --context=fra --namespace=public
--record=true
creationTimestamp: "2023-04-03T13:36:05Z"
name: pn-backend
namespace: public
resourceVersion: "227720878"
uid: eb76e588-b3a4-4299-bf85-ee1e6e818ada
spec:
clusterIP: 10.245.106.220
clusterIPs:
- 10.245.106.220
externalTrafficPolicy: Cluster
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: http
nodePort: 30569
port: 4000
protocol: TCP
targetPort: 4000
- name: ws
nodePort: 30073
port: 4001
protocol: TCP
targetPort: 4001
- name: grpc
nodePort: 30022
port: 9090
protocol: TCP
targetPort: 9090
- name: web
nodePort: 30754
port: 9091
protocol: TCP
targetPort: 9091
- name: web-ws
nodePort: 30693
port: 9092
protocol: TCP
targetPort: 9092
selector:
app: pn-backend
sessionAffinity: None
type: NodePort
status:
loadBalancer: {}
[nariman@notebook new-only-back]$ kubectl -n public get endpointslices.discovery.k8s.io pn-backend-h9fwc -o yaml
addressType: IPv4
apiVersion: discovery.k8s.io/v1
endpoints:
- addresses:
- 10.244.48.184
conditions:
ready: true
serving: true
terminating: false
nodeName: public-node-k8s-pool-fra1-jjfif
targetRef:
kind: Pod
name: pn-backend-748b569678-5sqcb
namespace: public
uid: 363edb7a-aa40-4468-ba30-5cfaf712262a
- addresses:
- 10.244.0.11
conditions:
ready: true
serving: true
terminating: false
nodeName: public-node-k8s-pool-fra1-jggse
targetRef:
kind: Pod
name: pn-backend-748b569678-9szzn
namespace: public
uid: 350bbdef-de2d-455d-8841-306337e8ad47
- addresses:
- 10.244.40.54
conditions:
ready: true
serving: true
terminating: false
nodeName: public-node-k8s-pool-fra1-jo28y
targetRef:
kind: Pod
name: pn-backend-748b569678-9hztz
namespace: public
uid: c82f3d93-9a50-4f7e-9ce7-cef6d55cf2ef
- addresses:
- 10.244.2.141
conditions:
ready: true
serving: true
terminating: false
nodeName: public-node-k8s-pool-fra1-j6x9c
targetRef:
kind: Pod
name: pn-backend-748b569678-gfzmd
namespace: public
uid: 03245b86-2905-4741-8674-7239efa3175c
- addresses:
- 10.244.4.89
conditions:
ready: true
serving: true
terminating: false
nodeName: public-node-k8s-pool-fra1-j6x98
targetRef:
kind: Pod
name: pn-backend-748b569678-pj9ph
namespace: public
uid: b54f1bcc-af6a-4e46-b7e6-7c923a3989c3
- addresses:
- 10.244.3.157
conditions:
ready: true
serving: true
terminating: false
nodeName: public-node-k8s-pool-fra1-j6x9u
targetRef:
kind: Pod
name: pn-backend-748b569678-k8vgf
namespace: public
uid: 6e83f356-836c-4872-b125-884e7f4c1d76
- addresses:
- 10.244.51.85
conditions:
ready: true
serving: true
terminating: false
nodeName: public-node-k8s-pool-fra1-jb9u4
targetRef:
kind: Pod
name: pn-backend-748b569678-mxtq4
namespace: public
uid: 60167a0c-90a7-400a-8797-8f743a99a751
- addresses:
- 10.244.52.54
conditions:
ready: true
serving: true
terminating: false
nodeName: public-node-k8s-pool-fra1-jb9c5
targetRef:
kind: Pod
name: pn-backend-748b569678-zvslm
namespace: public
uid: 47e7e6f2-8473-4ab9-86d8-2d87180c83f9
- addresses:
- 10.244.49.232
conditions:
ready: true
serving: true
terminating: false
nodeName: public-node-k8s-pool-fra1-jjy7i
targetRef:
kind: Pod
name: pn-backend-748b569678-6m87p
namespace: public
uid: 0462dc9d-bd8c-4048-b3d9-1464c3b617a1
- addresses:
- 10.244.0.163
conditions:
ready: true
serving: true
terminating: false
nodeName: public-node-k8s-pool-fra1-jggsa
targetRef:
kind: Pod
name: pn-backend-748b569678-rvhfc
namespace: public
uid: f111baaf-5701-400e-b696-49a4be3dc803
- addresses:
- 10.244.11.130
conditions:
ready: true
serving: true
terminating: false
nodeName: public-node-k8s-pool-fra1-jols6
targetRef:
kind: Pod
name: pn-backend-748b569678-fvk98
namespace: public
uid: 09393517-cc36-4315-82b5-7a80c804bbfc
- addresses:
- 10.244.24.99
conditions:
ready: true
serving: true
terminating: false
nodeName: public-node-k8s-pool-fra1-jols2
targetRef:
kind: Pod
name: pn-backend-748b569678-2b5fc
namespace: public
uid: e1985bea-331c-4235-9a16-9d9f7d9d01e9
- addresses:
- 10.244.47.216
conditions:
ready: true
serving: true
terminating: false
nodeName: public-node-k8s-pool-fra1-jjfiq
targetRef:
kind: Pod
name: pn-backend-748b569678-5h56d
namespace: public
uid: f246aeaf-0408-450a-822c-28e861eea19b
- addresses:
- 10.244.39.214
conditions:
ready: true
serving: true
terminating: false
nodeName: public-node-k8s-pool-fra1-jo2ns
targetRef:
kind: Pod
name: pn-backend-748b569678-t99pc
namespace: public
uid: 8508225f-8cf2-48e6-bdff-9304964c82bf
- addresses:
- 10.244.37.252
conditions:
ready: true
serving: true
terminating: false
nodeName: public-node-k8s-pool-fra1-jol98
targetRef:
kind: Pod
name: pn-backend-748b569678-7dcj8
namespace: public
uid: 21e8293b-4a51-4701-9c42-d45619919c95
- addresses:
- 10.244.43.186
conditions:
ready: true
serving: true
terminating: false
nodeName: public-node-k8s-pool-fra1-jo28r
targetRef:
kind: Pod
name: pn-backend-748b569678-h45nr
namespace: public
uid: 17a9ff72-b098-4f7f-a17e-a154a987113a
- addresses:
- 10.244.42.195
conditions:
ready: true
serving: true
terminating: false
nodeName: public-node-k8s-pool-fra1-jo28j
targetRef:
kind: Pod
name: pn-backend-748b569678-84vqk
namespace: public
uid: c3db49b2-a931-4647-ac50-0ead08719f60
- addresses:
- 10.244.51.146
conditions:
ready: true
serving: true
terminating: false
nodeName: public-node-k8s-pool-fra1-jb9ui
targetRef:
kind: Pod
name: pn-backend-748b569678-bfk8d
namespace: public
uid: 30f9461c-69f6-43b9-9be2-8489e8bcd13b
- addresses:
- 10.244.24.217
conditions:
ready: true
serving: true
terminating: false
nodeName: public-node-k8s-pool-fra1-jolii
targetRef:
kind: Pod
name: pn-backend-748b569678-bxvxn
namespace: public
uid: 94b6cfdb-dbe5-48e0-ae06-a333be26e9c5
kind: EndpointSlice
metadata:
annotations:
endpoints.kubernetes.io/last-change-trigger-time: "2024-04-24T22:12:00Z"
creationTimestamp: "2023-04-03T13:36:05Z"
generateName: pn-backend-
generation: 40139
labels:
endpointslice.kubernetes.io/managed-by: endpointslice-controller.k8s.io
kubernetes.io/service-name: pn-backend
name: pn-backend-h9fwc
namespace: public
ownerReferences:
- apiVersion: v1
blockOwnerDeletion: true
controller: true
kind: Service
name: pn-backend
uid: eb76e588-b3a4-4299-bf85-ee1e6e818ada
resourceVersion: "249331656"
uid: cb4a980f-e75f-4dfe-a26f-aa472daa0b93
ports:
- name: grpc
port: 9090
protocol: TCP
- name: web
port: 9091
protocol: TCP
- name: web-ws
port: 9092
protocol: TCP
- name: ws
port: 4001
protocol: TCP
- name: http
port: 4000
protocol: TCP
Hi @brianehlert
Than you for your previous response
Is it not possible to add health check if I don't use Nginx Plus ?
Warning Rejected 28s nginx-ingress-controller VirtualServer public/pn-front-prod-arbitrum-nova-rpc was rejected with error: spec.upstreams[0].healthCheck: Forbidden: active health checks are only supported in NGINX Plus
[nariman@notebook nginx-health]$ kubectl -n public get virtualserver pn-front-prod-arbitrum-nova-rpc
NAME STATE HOST IP PORTS AGE
pn-front-prod-arbitrum-nova-rpc Invalid arbitrum-nova-rpc.example.com 41d
Passive health checks are always present. But Active health checks are a capability that is specific to NGINX Plus. https://docs.nginx.com/nginx/admin-guide/load-balancer/http-health-check/
By default, NGINX Ingress Controller won't add pods to the service upstream group until the pod reports ready. So, the alternative to using the enterprise version of this project is to improve the ready probe on your service pods.
Hello,
Yes my deployment is configure with live and read probe Also as you mentioned passive health is automatically enabled by nginx virtual host template But I am faced this problem after pod is not present And nginx try to send traffic to pod which died 10-20 minute before
[nariman@notebook nginx-ingress]$ kubectl -n public get deployments.apps pn-backend -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: pn-backend
tags.datadoghq.com/service: pn-backend
name: pn-backend
namespace: public
spec:
selector:
matchLabels:
app: pn-backend
strategy:
rollingUpdate:
maxSurge: 10%
maxUnavailable: 10%
type: RollingUpdate
template:
metadata:
annotations:
ad.datadoghq.com/pn-backend.logs: '[{"source":"pn-backend","service":"pn-backend","auto_multi_line_detection":true}]'
creationTimestamp: null
labels:
app: pn-backend
tags.datadoghq.com/env: prod-fra
tags.datadoghq.com/service: pn-backend
spec:
containers:
- name: pn-backend
image: public/pn-backend:fe1db1c
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /api/healthcheck
port: 4000
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 3
successThreshold: 1
timeoutSeconds: 3
ports:
- containerPort: 4000
protocol: TCP
- containerPort: 9090
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /api/healthcheck
port: 4000
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 3
resources:
limits:
cpu: "12"
memory: 16Gi
requests:
cpu: "10"
memory: 8Gi
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /var/run/datadog
name: apmsocketpath
- mountPath: /app/geo_config.json
name: cluster-configs-volume
readOnly: true
subPath: geo_config.json
dnsPolicy: ClusterFirst
imagePullSecrets:
- name: docker-registry
initContainers:
- command:
- /bin/sh
- -c
- |
sysctl -w net.core.somaxconn=64000
sysctl -w net.ipv4.ip_local_port_range="1024 65535"
image: busybox
imagePullPolicy: Always
name: init-sysctl
resources: {}
securityContext:
privileged: true
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
terminationGracePeriodSeconds: 30
volumes:
- hostPath:
path: /var/run/datadog/
type: ""
name: apmsocketpath
- name: cluster-configs-volume
secret:
defaultMode: 420
secretName: cluster-configs-f9c45ad972b6b8559cdc924581631d693f53d5d0
The deployment doesn't give us much information to assist with. We would need you to share your configuration resources. The VirtualServer, VirtualServerRoute, TransportServer, or Ingress
If a pod of a service no longer exists, it should be removed from the ingress controller upstream group for that service. Unless there is a configuration error (such as through snippets or customizations) that is preventing NGINX from being updated.
Hi, thank you for fast response If i forgot something please inform me.
- We installed our nginx via helm chart
[nariman@notebook ~]$ helm ls -n nginx
NAME NAMESPACE REVISION UPDATED STATUS CHART APP VERSION
ingress nginx 19 2024-03-29 12:17:00.734183664 +0400 +04 deployed nginx-ingress-1.1.0 3.4.0
- Nginx ConfigMap
[nariman@notebook ~]$ kubectl -n nginx get cm ingress-nginx-ingress -o yaml
apiVersion: v1
data:
client-max-body-size: 100m
http2: "true"
log-format: date="$time_iso8601" status=$status request_completion=$request_completion
msec=$msec connections_active=$connections_active connections_reading=$connections_reading
connections_writing=$connections_writing connections_waiting=$connections_waiting
connection=$connection connection_requests=$connection_requests connection_time=$connection_time
client=$remote_addr method=$request_method request="$request" request_length=$request_length
status=$status bytes_sent=$bytes_sent body_bytes_sent=$body_bytes_sent referer=$http_referer
user_agent="$http_user_agent" upstream_addr=$upstream_addr upstream_status=$upstream_status
request_time=$request_time upstream_response_time=$upstream_response_time upstream_connect_time=$upstream_connect_time
upstream_header_time=$upstream_header_time request_body="$request_body host="$host"
user_ip="$http_x_forwarded_for"
log-format-escaping: json
proxy-buffering: "false"
proxy-request-buffering: "off"
redirect-to-https: "true"
ssl_buffer_size: 4k
kind: ConfigMap
metadata:
annotations:
meta.helm.sh/release-name: ingress
meta.helm.sh/release-namespace: nginx
creationTimestamp: "2024-03-26T14:37:04Z"
labels:
app.kubernetes.io/instance: ingress
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: nginx-ingress
app.kubernetes.io/version: 3.4.0
helm.sh/chart: nginx-ingress-1.1.0
name: ingress-nginx-ingress
namespace: nginx
resourceVersion: "222418677"
uid: 958b1e7e-d36e-44cf-bfc1-d5aee88b767d
- Service in nginx namespace, we do not use LoadBalancer type, because before Nginx we use CloudFlare tunnel
[nariman@notebook ~]$ kubectl -n nginx get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-ingress-controller NodePort 10.245.73.204 <none> 80:32098/TCP,443:32685/TCP 43d
ingress-nginx-ingress-prometheus-service ClusterIP None <none> 9113/TCP 43d
- Nginx deployment
[nariman@notebook ~]$ kubectl -n nginx get deployments.apps ingress-nginx-ingress-controller -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "10"
meta.helm.sh/release-name: ingress
meta.helm.sh/release-namespace: nginx
creationTimestamp: "2024-03-26T14:37:05Z"
generation: 158
labels:
app.kubernetes.io/instance: ingress
app.kubernetes.io/managed-by: Helm
app.kubernetes.io/name: nginx-ingress
app.kubernetes.io/version: 3.4.0
helm.sh/chart: nginx-ingress-1.1.0
name: ingress-nginx-ingress-controller
namespace: nginx
resourceVersion: "250051917"
uid: 6cda1863-0a5a-4b90-b419-74e6f26540b2
spec:
progressDeadlineSeconds: 600
replicas: 10
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/instance: ingress
app.kubernetes.io/name: nginx-ingress
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
kubectl.kubernetes.io/restartedAt: "2024-03-29T10:34:14+04:00"
prometheus.io/port: "9113"
prometheus.io/scheme: http
prometheus.io/scrape: "true"
creationTimestamp: null
labels:
app.kubernetes.io/instance: ingress
app.kubernetes.io/name: nginx-ingress
spec:
automountServiceAccountToken: true
containers:
- args:
- -nginx-plus=false
- -nginx-reload-timeout=60000
- -enable-app-protect=false
- -enable-app-protect-dos=false
- -nginx-configmaps=$(POD_NAMESPACE)/ingress-nginx-ingress
- -ingress-class=nginx
- -health-status=false
- -health-status-uri=/nginx-health
- -nginx-debug=false
- -v=1
- -nginx-status=true
- -nginx-status-port=8080
- -nginx-status-allow-cidrs=127.0.0.1
- -report-ingress-status
- -enable-leader-election=true
- -leader-election-lock-name=nginx-ingress-leader
- -enable-prometheus-metrics=true
- -prometheus-metrics-listen-port=9113
- -prometheus-tls-secret=
- -enable-service-insight=false
- -service-insight-listen-port=9114
- -service-insight-tls-secret=
- -enable-custom-resources=true
- -enable-snippets=false
- -include-year=false
- -disable-ipv6=false
- -enable-tls-passthrough=false
- -enable-cert-manager=false
- -enable-oidc=false
- -enable-external-dns=false
- -default-http-listener-port=80
- -default-https-listener-port=443
- -ready-status=true
- -ready-status-port=8081
- -enable-latency-metrics=true
- -ssl-dynamic-reload=true
env:
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
image: nginx/nginx-ingress:3.4.0
imagePullPolicy: IfNotPresent
name: nginx-ingress
ports:
- containerPort: 80
name: http
protocol: TCP
- containerPort: 443
name: https
protocol: TCP
- containerPort: 9113
name: prometheus
protocol: TCP
- containerPort: 8081
name: readiness-port
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /nginx-ready
port: readiness-port
scheme: HTTP
periodSeconds: 1
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: "2"
memory: 2Gi
requests:
cpu: "1"
memory: 512Mi
securityContext:
allowPrivilegeEscalation: false
capabilities:
add:
- NET_BIND_SERVICE
drop:
- ALL
readOnlyRootFilesystem: false
runAsNonRoot: true
runAsUser: 101
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
seccompProfile:
type: RuntimeDefault
serviceAccount: ingress-nginx-ingress
serviceAccountName: ingress-nginx-ingress
terminationGracePeriodSeconds: 30
status:
availableReplicas: 10
conditions:
- lastTransitionTime: "2024-03-26T14:37:05Z"
lastUpdateTime: "2024-03-29T06:34:21Z"
message: ReplicaSet "ingress-nginx-ingress-controller-565c6849d5" has successfully
progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
- lastTransitionTime: "2024-04-10T16:13:10Z"
lastUpdateTime: "2024-04-10T16:13:10Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 158
readyReplicas: 10
replicas: 10
updatedReplicas: 10
Follow manifest related our service
- VirtualServer manifest
[nariman@notebook ~]$ kubectl -n public get virtualservers.k8s.nginx.org pn-front-prod-arbitrum-sepolia-rpc
NAME STATE HOST IP PORTS AGE
pn-front-prod-arbitrum-sepolia-rpc Valid arbitrum-sepolia-rpc.public.com 43d
[nariman@notebook ~]$ kubectl -n public get virtualservers.k8s.nginx.org pn-front-prod-arbitrum-sepolia-rpc -o yaml
apiVersion: k8s.nginx.org/v1
kind: VirtualServer
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"k8s.nginx.org/v1","kind":"VirtualServer","metadata":{"annotations":{},"name":"pn-front-prod-arbitrum-sepolia-rpc","namespace":"public"},"spec":{"host":"arbitrum-sepolia-rpc.public.com","routes":[{"action":{"pass":"frontend"},"matches":[{"action":{"redirect":{"code":301,"url":"https://arbitrum-sepolia-rpc.public.com/"}},"conditions":[{"value":"GET","variable":"$request_method"}]}],"path":"/api/metrics"},{"action":{"pass":"frontend"},"matches":[{"action":{"pass":"frontend"},"conditions":[{"value":"GET","variable":"$request_method"}]}],"path":"/api"},{"action":{"pass":"frontend"},"matches":[{"action":{"pass":"frontend"},"conditions":[{"value":"GET","variable":"$request_method"}]}],"path":"/favicon.ico"},{"action":{"pass":"frontend"},"matches":[{"action":{"pass":"frontend"},"conditions":[{"value":"GET","variable":"$request_method"}]}],"path":"/platforms"},{"action":{"pass":"frontend"},"matches":[{"action":{"pass":"frontend"},"conditions":[{"value":"GET","variable":"$request_method"}]}],"path":"/_next"},{"action":{"pass":"backend"},"matches":[{"action":{"pass":"backend"},"conditions":[{"header":"Upgrade","value":"websocket"}]},{"action":{"proxy":{"rewritePath":"/arbitrum-sepolia","upstream":"frontend"}},"conditions":[{"value":"GET","variable":"$request_method"}]}],"path":"/"}],"server-snippets":"proxy_request_buffering off;\nssl_buffer_size 4k;\n","tls":{"secret":"public.com"},"upstreams":[{"name":"backend","port":4000,"service":"pn-backend"},{"name":"frontend","port":3000,"service":"pn-frontend"}]}}
creationTimestamp: "2024-03-26T14:42:08Z"
generation: 1
name: pn-front-prod-arbitrum-sepolia-rpc
namespace: public
resourceVersion: "222416877"
uid: e616e0dc-3433-4be4-807d-00786e8a217d
spec:
host: arbitrum-sepolia-rpc.public.com
routes:
- action:
pass: frontend
matches:
- action:
redirect:
code: 301
url: https://arbitrum-sepolia-rpc.publicn.com/
conditions:
- value: GET
variable: $request_method
path: /api/metrics
- action:
pass: frontend
matches:
- action:
pass: frontend
conditions:
- value: GET
variable: $request_method
path: /api
- action:
pass: frontend
matches:
- action:
pass: frontend
conditions:
- value: GET
variable: $request_method
path: /favicon.ico
- action:
pass: frontend
matches:
- action:
pass: frontend
conditions:
- value: GET
variable: $request_method
path: /platforms
- action:
pass: frontend
matches:
- action:
pass: frontend
conditions:
- value: GET
variable: $request_method
path: /_next
- action:
pass: backend
matches:
- action:
pass: backend
conditions:
- header: Upgrade
value: websocket
- action:
proxy:
rewritePath: /arbitrum-sepolia
upstream: frontend
conditions:
- value: GET
variable: $request_method
path: /
server-snippets: |
proxy_request_buffering off;
ssl_buffer_size 4k;
tls:
secret: public.com
upstreams:
- name: backend
port: 4000
service: pn-backend
- name: frontend
port: 3000
service: pn-frontend
status:
message: 'Configuration for public/pn-front-prod-arbitrum-sepolia-rpc was added
or updated '
reason: AddedOrUpdated
state: Valid
- virtualserverroutes
[nariman@notebook ~]$ kubectl -n public get virtualserverroutes.k8s.nginx.org
No resources found in public namespace.
- transportservers
[nariman@notebook ~]$ kubectl -n public get transportservers.k8s.nginx.org
No resources found in public namespace.
- ingress
[nariman@notebook ~]$ kubectl -n public get ingress
No resources found in public namespace.
[nariman@notebook ~]$ kubectl -n public get svc
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
pn-backend NodePort 10.245.106.220 <none> 4000:30569/TCP,4001:30073/TCP,9090:30022/TCP,9091:30754/TCP,9092:30693/TCP 401d
pn-connections-broker NodePort 10.245.65.60 <none> 8888:30605/TCP,9999:32137/TCP 71d
pn-cron NodePort 10.245.206.66 <none> 5005:32416/TCP 196d
pn-frontend NodePort 10.245.253.158 <none> 3000:31404/TCP 401d
pn-internal-stats NodePort 10.245.116.36 <none> 4000:30191/TCP,4444:32162/TCP 174d
Hi all, Thanks for your previous responses
Do I need to provide anything else ? Some manifests I added in the first comments. As I mentioned earlier, nginx is trying to send requests to IP addresses of pods that are no longer alive. For example: We are updating our backend component - and want to update our deployment to a new version of the container image, as a result of updating to a new version of the image, all our pods are restarted and new pods are started with new IP addresses, but nginx still tries to send traffic to an IP that no longer exists.
@ibadullaev-inc4 are you able to confirm if this occurs only during a rolling upgrade of the pn-backend service?
Note, you have server-snippets configured, yet -enable-snippets=false is set.
@ibadullaev-inc4 before sending requests to the service, did the backend service deployment complete and the nginx reload finish?
Hello @pdabelf5
before sending requests to the service, did the backend service deployment complete ?
When I say that we are restarting the service, I mean we are deploying a new version of our service. After we update our deployment, our pods sequentially terminate and new ones appear, which have the new image version. Yes, we observe this behavior after 30-40 minutes. That is, the deployment finishes updating, then 30 minutes pass, and we see that this pod has been gone for 30 minutes, but nginx is still trying to send traffic to it.
and the nginx reload finish?
I didn't understand this part of the question: we don't restart nginx when we deploy a new version of our backend.
are you able to confirm if this occurs only during a rolling upgrade of the pn-backend service?
Yes, we observe this issue only in this case.
Note, you have server-snippets configured, yet -enable-snippets=false is set.
Do you mean that we should switch this parameter to True?
Note: When I manually delete a pod using the command "kubectl delete pods backend-xxxxxxx" which has the IP, for example: X.X.X.X, I see that nginx removes this IP address from its upstream configuration. This means nginx behaves correctly and stops passive monitoring for this IP.
But when updating the backend service, nginx most likely does not remove the old IPs from its configuration and continues to send traffic to them.
Hello @ibadullaev-inc4 Thank you for clarifying the behaviour you are seeing.
In order for
server-snippets: |
proxy_request_buffering off;
ssl_buffer_size 4k;
to take effect, -enable-snippets=true needs to be set in your manifest or controller.enableSnippets: true in the helm values file.
and the nginx reload finish?
Nginx should reload when the upstream pod ip's in the backend service are updated & the upstream config should contain the current list of pod ip's.
I am trying to replicate the issue you are seeing. Is the configuration you have provided the smallest example configuration that results in the problem you have encountered? Is there anything I might need when replicating the issue, i.e. websockets or grpc?
The example you provided suggests you are using 3.4.0, have you tried 3.4.3 or 3.5.1 and seen the same issue?
Hello @pdabelf5 I want to thank you for your support and quick response.
-enable-snippets=true
Yes, we forgot to enable this setting. Thank you for your attention.
Nginx should reload when the upstream pod ip's in the backend service are updated & the upstream config should contain the current list of pod ip's.
Oh, that's very strange. Are you sure I need to restart nginx every time I update my backend service (deployment)? Currently, we are using Istio and we don't restart it when deploying a new version. Should the reload process happen automatically, or do I need to initiate it somehow?
I am trying to replicate the issue you are seeing. Is the configuration you have provided the smallest example configuration that results in the problem you have encountered? Is there anything I might need when replicating the issue, i.e. websockets or grpc?
"We also use gRPC and WebSockets in our configuration, but I'm not sure if they could influence or be the source of the problem. I think you can try without them, as we see that the timeout issue is related to the HTTP protocol."
The example you provided suggests you are using 3.4.0, have you tried 3.4.3 or 3.5.1 and seen the same issue?
We only tried version 3.4.0.
@ibadullaev-inc4
Should the reload process happen automatically, or do I need to initiate it somehow?
Apologies if I implied the nginx reload was something you needed to perform, it should happen automatically when nginx config is updated by the controller.
I will let you know how things go trying to replicate the issue as soon as I have something to tell.
@ibadullaev-inc4
Currently, we are using Istio and we don't restart it when deploying a new version.
Can you expand on how you are using Istio in your setup? Is it separate? Is it a sidecar within NGINX Ingress controller? TY
@ibadullaev-inc4 I put together a test that deploys a test application, updates the test application whilst performing requests while it is updating. I'm afraid I wasn't able to reproduce the problem you have encountered.
