kubernetes
Hairpin connection failed on Service type-LoadBalancer NLB with internal scheme
What happened:
Service-type LoadBalancer NLB with internal scheme are failing to expose services in hairpin scenarios: when the client is hosted in the same node as server/backend/target.
graph TD
subgraph Instance
OVN
NIC["Network Card (ENI)"]
end
subgraph OVN["OVN-Kubernetes network plugin"]
direction TB
A[Client Pod]
C[Backend Pod]
end
B["Service NLB-internal (target instance)"]
A -.->NIC -.-> B -.-> NIC -.-> C
This happens in the default target type (instance) on the Service-type NLB on internal scheme (default scheme). CCM by default enables the target attribute Preserve client IP addresses, which would be the root cause of this problem. The proxy is also off by default, and CCM does not expose API to customize both attributes.
Service object:
$ oc get service/lbconfig-test-hp-nlb-int -o yaml -n cloud-provider-aws-9871
apiVersion: v1
kind: Service
metadata:
annotations:
aws-load-balancer-backend-protocol: http
aws-load-balancer-ssl-ports: https
service.beta.kubernetes.io/aws-load-balancer-internal: "true"
service.beta.kubernetes.io/aws-load-balancer-target-node-labels: kubernetes.io/hostname=ip-10-0-4-233.ec2.internal
service.beta.kubernetes.io/aws-load-balancer-type: nlb
name: lbconfig-test-hp-nlb-int
namespace: cloud-provider-aws-9871
spec:
allocateLoadBalancerNodePorts: true
clusterIP: 172.30.143.233
clusterIPs:
- 172.30.143.233
externalTrafficPolicy: Cluster
internalTrafficPolicy: Cluster
ipFamilies:
- IPv4
ipFamilyPolicy: SingleStack
ports:
- name: http
nodePort: 31419
port: 80
protocol: TCP
targetPort: 8080
type: LoadBalancer
What you expected to happen:
Hairpin connection works on service type NLB with internal scheme when the target type is instance by allowing customization of target group attributes:
- Preserve client IP addresses
- Proxy protocol v2
We need to be able to have control of both attributes so we can keep working other features like: access logs, access lists, session persistence, balancing, and anything else that depends on the source address.
How to reproduce it (as minimally and precisely as possible):
- Select a node to deploy the client and server
- Create a backend server pod with affinity to the selected node
- Create a Service type-LoadBalancer NLB (annotation
service.beta.kubernetes.io/aws-load-balancer-type: nlb) with scheme internal (annotationservice.beta.kubernetes.io/aws-load-balancer-internal: "true") - Deploy a client making requests to the Load Balancer address
- Expected the request to fail
Example scenarios are implemented on PR https://github.com/kubernetes/cloud-provider-aws/pull/1161.
Anything else we need to know?:
The network plugin is OVN Kubernetes, the overlay network have it's own CIDR (does not share with VPC CIDR), and source/dest IPs of pods leaving the nodes would be the instance primary IP address.
Environment:
- Kubernetes version (use
kubectl version): v1.32.4 - Cloud provider or hardware configuration: AWS
- OS (e.g. from /etc/os-release): Red Hat Enterprise Linux CoreOS 9.6
- Kernel (e.g.
uname -a): 5.14.0-570.17.1.el9_6.x86_64 - Install tools: openshift-install
- Others:
/kind bug
This issue is currently awaiting triage.
If cloud-provider-aws contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.
The triage/accepted label can be added by org members by writing /triage accepted in a comment.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
