John Howard

I can also reproduce pretty easily. Thanks for the detailed issue :+1:

This is more workaround than fix since it requires NET_ADMIN on all pods, but the TPROXY mode (sidecar.istio.io/interceptionMode=TPROXY) annotation does fix this

I used https://github.com/howardjohn/istio/commit/75a67eb79c6ebf53850840f47cf4d93a52b46a00 to reproduce. Basically `client --flood svc:port1` then eventually `client svc:port2` fails. Some ideas... 1. eBPF redirection. https://github.com/merbridge/merbridge/ might just automatically solve this today, untested 2. TPROXY -...

FWIW I tried to see if we can avoid NET_ADMIN for TPROXY by using eBPF to set the IP_TRANSPARENT option; it almost works but the bpf has a limited set...

Would be interesting to see if ebpf solutions (https://istio.io/latest/blog/2022/merbridge/, etc) resolve this

One note: was browsing some eBPF network code, found https://github.com/cilium/cilium/blob/eddd2b3422cb8796900653595a267e8c7048c411/bpf/lib/proxy.h#L138. Suggests we could possibly unset source port when we set the dport and have it re-assigned in some cases

Nice find! Going to do some more research so I am up to speed but looks super promising. Looks like cilium does similar (https://docs.cilium.io/en/stable/operations/system_requirements/#requirements-for-l7-and-fqdn-policies) - they only do it without...

> We do require network permissions with our tproxy because it can spoof the source IP, but that's the second part and we can decouple it. Can you use TPROXY...

There is no setting ip_early_demux in any Istio currently. I am not sure if ambient is impacted by this; it plausible would not be due to TPROXY usage

Do to the excessive permissions required for TPROXY I don't consider it a particularly viable path forward