projectcalico
After migrating to Calico from Flannel, X-Forwarded-For headers set by the front proxy container are invalid
My on-prem cluster uses MetalLB and a front proxy (Envoy) within a pod. Previously, when using Flannel, the front proxy was able to see the original source IP address because externalTrafficPolicy: Local is set on the Kubernetes service for the front proxy. Since switching to Calico, the front proxy instead reports either a 10.244.0.0/16 or a 169.254.0.0/16 IP in its X-Forwarded-For header.
Expected Behavior
The Flannel migration should work the same as it did before.
Current Behavior
Calico does not appear to properly report the originating IP address to the Envoy container.
Possible Solution
I'm not sure how to fix this; I'm not familiar enough to know which Calico setting I even need to use here, despite reading docs — I can't make heads or tails of ipipMode and vxlanMode.
Steps to Reproduce (for bugs)
Sample service:
apiVersion: v1
kind: Service
metadata:
name: switchboard
spec:
type: LoadBalancer
loadBalancerIP: 10.0.0.100
externalTrafficPolicy: Local # "pods can see the real source IP address of incoming connections."
ports:
- name: http
port: 80
targetPort: 80
And here is my pool:
apiVersion: projectcalico.org/v3
items:
- apiVersion: projectcalico.org/v3
kind: IPPool
metadata:
creationTimestamp: "2023-07-10T20:54:28Z"
name: default-ipv4-ippool
resourceVersion: "123754290"
uid: d11ad29c-938e-4719-acfd-c6dc4627e762
spec:
allowedUses:
- Workload
- Tunnel
blockSize: 26
cidr: 10.244.0.0/16
ipipMode: Always
natOutgoing: true
nodeSelector: all()
vxlanMode: Never
kind: IPPoolList
metadata:
resourceVersion: "123865121"
Context
This prevents me from using IP-based security in my network. Previously I had whitelisted the internal 192.168.1.0/24 range to allow my home network secure access to login to certain selfhosted services.
Your Environment
- Calico version: latest (?) just installed
- Orchestrator version (e.g. kubernetes, mesos, rkt): 1.27.3
- Operating System and version: Linux
Hi @zaneclaes , based on my understanding and the IPPool you shared, I would expect your pods to be in the 10.244.0.0/16 range so I would have thought that those IPs would have been expected. It might be that I am misunderstanding the requests you are making when you see the logs in your front proxy. Can you also give us details on how you made those requests with the IPPool and service provided? You have also mentioned the 169.254.0.0/16 and 192.168.1.0/24 ranges, what do those represent?
The point is that the IPs I see should NOT be anything at all related to Calico. The goal is that the front-proxy properly sets X-Forwarded-For to the IP of the computer which originated the request, so that I may use the header in downstream services which depend on it as a mode of authentication. The 10.244.x.x is the internal IP range used by Kubernetes/Calico (as shown by the pods themselves), the 169.254.x.x is apparently related to Calico (the docs say its a router?) and did not exist before I migrated from Flannel to Calico. The 192.168.x.x is my home network.
In short, if I am on my home WiFi and open a website served by the front proxy, the X-forwarded-for should be set to 192.168.x.x and not one of those other two ranges created by Calico. And if I open it from my phone outside the house (since I do expose my house to the outside world), it should be set to whatever IP my cell provider assigns. Thus I can properly use the 192.168.0.0/16 subnet to authenticate local on-prem traffic.
Can you also give us details on how you made those requests with the IPPool and service provided?
In case this wasn't clear...
I just open up a web browser and type the URL. When on my home network, the router responds to the DNS lookup with the IP 10.0.0.100 (when outside the home network, it port-forwards 80 and 443 to this internal IP address). Thus the request is received MetalLB (per the service definition in the post) and sent to the front-proxy pod. The front proxy (envoy) is running as a DaemonSet, but nodeAffinity restricted to those machines which serve as ingress...
@zaneclaes could you share some more details about your setup? Say, a simple diagram of the topology (how many nodes, how they connect to each other and your router, with the IPs each one has, the routes on each node and how you do routing on your cluster/network), the exact calico version (you can get this from kubectl describe pod on a calico-node pod), etc...
If you could share your flannel config and how exactly you migrated from calico (did you just create a new "blank" cluster with calico and spun up your services on it? something else?), that could help.
Re: IPIP vs. VXLAN, do you have any specific questions? Did you use this doc https://docs.tigera.io/calico/latest/networking/configuring/vxlan-ipip? We'd definitely appreciate pointers on what this doc might fail to explain clearly...
Thanks!
Hi @zaneclaes, sorry, for the late reply. In addition to the details @coutinhop shared above, what is a bit confusing about your use case is that envoy is seeing the Calico pod CIDR at all. I would have expected that envoy would be the entry point into your cluster and so your source IP should be preserved in the X-Forward-For header since Calico should not be modifying HTTP headers. The only way I would have expected you to see any other IP is if the node that your request landed on had none of the envoy pods on it, so traffic is sent to a separate node and NAT'ed but even in this case, you would be seeing an IP from your node CIDR, not your pod CIDR. Is it possible that traffic is landing at a separate pod before it hits your envoy pods?
@coutinhop I can try to diagram it... but I think this is severely overcomplicating things. There are only 2 relevant pieces AFAICT: (1) MetalLB (2) Envoy. When a request hits Kubernetes MetalLB and forwards it to Envoy, the IP is incorrect. Everything else is superficial. The upstream (WAN, etc.) has no impact, and the downstream (Home Assistant) happens after the problem:
There are exactly 2 nodes, each running on the "infra" VLAN (10.0.0.x) with static IPs:
- The master node (
conductor) runs Envoy - The second node (
nixie-beast) runs the Home Assistant application
Here is kubectl get nodes -owide:
conductor Ready control-plane,master 2y59d v1.27.3 10.0.0.10 <none> Ubuntu 20.04.6 LTS 5.4.0-153-generic containerd://1.6.21
nixie-beast Ready <none> 2y27d v1.27.3 10.0.0.15 <none> Ubuntu 20.04.6 LTS 5.15.0-76-generic containerd://1.6.21
Here is the Envoy process and Home Assistant process:
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
switchboard-ct2vv 1/1 Running 3 16d 10.244.83.73 conductor <none> <none>
home-assistant-0 1/1 Running 1 11d 10.244.218.74 nixie-beast <none> <none>
Here's the full describe on a Calico node.
As for flannel, I had originally installed it when I created the cluster by doing an install of https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml and the flag --pod-network-cidr=10.96.0.0/16 during kubeadm init. I did no other customization, and didn't think about Flannel at all until I tried to upgrade to Calico via the instructions here.
Finally, IPIP vs. VXLAN, I'm simply unfamiliar with these two terms and have not taken the time to find some external guide as to how they each behave. Perhaps this is out of scope for the documentation, but all the "encapsulation types" section really tells me is that there is a difference in overhead. It vaguely sounds like encapsulation might be related to my issue, but I'm not sure... I don't have a good sense of what problems the wrong method would create.
@mgleung I think MetalLB is what you are looking for (see the first sentence of the bug report). Here is the service description, which is patched with this layer2 reservation for the 10.0.0.100-200 range on the home VLAN for this traffic. Again, this worked fine with Flannel... no changes were made to MetalLB when switching to Calico.
@zaneclaes sorry for the long delay. Are you still seeing this issue or have you moved on from it? If you still need help, we'd need more info...
Yes, this is still happening @coutinhop ... I have been awaiting a reply to the additional info which I already provided... so I'm not sure what you're referring to.
Calico does not appear to properly report the originating IP address to the Envoy container.
Calico doesn't report anything to the Envoy container, the Envoy container should be setting this based on the source IP address on the connection that it received, so we should follow the path of the connection to see where along that path the source IP is being modified.
Finally, IPIP vs. VXLAN, I'm simply unfamiliar with these two terms and have not taken the time to find some external guide as to how they each behave.
I don't think IPIP vs VXLAN is going to be the issue here, because assuming externalTrafficPolicy: Local is behaving correctly, then the traffic in question will never actually be encapsulated (it arrives on the same node it needs to be one, and encap is only used for cross node comms).
More likely we're looking for some sort of NAT. For example, if externalTrafficPolicy: Local isn't behaving correctly, then you will see packets arrive on nodes and be forwarded to another node, at which point SNAT will occur and you would see behavior like you're describing.
Just to confirm, you're using MetalLB's L2 service implementation and not Calico's BGP service IP advertisement? I'm not very familiar with MetalLB's L2 implementation, but it's possible that something there is interacting with how Calico functions.
One place to look would be to monitor if any SNAT or MASQ` rules are being hit in iptables, via a command like this on your ingress nodes while they are receiving traffic:
iptables-save -c | grep SNAT
iptables-save -c | grep MASQ
the 169.254.x.x is apparently related to Calico (the docs say its a router?)
That range is the IPv4 link-local range, which Calico does "use" in some very limited capacity, but should actually shows up on any packets. I am struggling to think how this IP showing up in the header would be related to Calico.
