istio
istioctl proxy-config does not work on EKS IPv6 clusters
Is this the right place to submit this?
- [X] This is not a security vulnerability or a crashing bug
- [X] This is not a question about how to use Istio
Bug Description
The important detail here I believe is that I'm using AWS EKS 1.29 IPv6 with their weird IPv4 link-local NAT egress.
Related analysis from @howardjohn (thanks!) https://github.com/istio/istio/issues/49476#issuecomment-1959864574.
I am able to shell into a pod and run curl http://localhost:15000/config_dump however all istioctl pc commands fail.
❯ istioctl pc l myapp-web-5f5b6c695f-n97jz
2024-02-22T17:20:23.339406Z error klog an error occurred forwarding 60619 -> 15000: error forwarding port 15000 to pod 23c62f702462c62a8fd4ee632038302b4bc7e4a235264a3033d806bb51d26a41, uid : failed to execute portforward in network namespace "/var/run/netns/cni-b8d29c04-fabf-ba84-4e72-5bb792cbd579": failed to connect to localhost:15000 inside namespace "23c62f702462c62a8fd4ee632038302b4bc7e4a235264a3033d806bb51d26a41", IPv4: dial tcp4 127.0.0.1:15000: connect: connection refused IPv6 dial tcp6: address localhost: no suitable
address found
2024-02-22T17:20:23.341278Z error port forward failed: lost connection to pod
Error: failed to execute command on myapp-web-5f5b6c695f-n97jz.myapp sidecar: failure running port forward process: Get "http://localhost:60619/config_dump": EOF
With logging enabled:
❯ istioctl pc l myapp-web-6dc987cfb8-vw5nx --vklog 9
2024-02-22T17:11:18.019274Z info klog curl -v -XGET -H "Accept: application/vnd.kubernetes.protobuf,application/json" -H "User-Agent: istioctl/1.20.3" 'https://REDACTED.gr7.us-east-2.eks.amazonaws.com/api/v1/namespaces/myapp/pods/myapp-web-6dc987cfb8-vw5nx'
2024-02-22T17:11:18.515508Z info klog HTTP Trace: DNS Lookup for REDACTED.gr7.us-east-2.eks.amazonaws.com resolved to [{10.34.7.111 } {10.34.105.167 }]
2024-02-22T17:11:18.618212Z info klog HTTP Trace: Dial to tcp:10.34.7.111:443 succeed
2024-02-22T17:11:18.870643Z info klog GET https://REDACTED.gr7.us-east-2.eks.amazonaws.com/api/v1/namespaces/myapp/pods/myapp-web-6dc987cfb8-vw5nx 200 OK in 851 milliseconds
2024-02-22T17:11:18.870997Z info klog HTTP Statistics: DNSLookup 75 ms Dial 101 ms TLSHandshake 115 ms ServerProcessing 132 ms Duration 851 ms
2024-02-22T17:11:18.871016Z info klog Response Headers:
2024-02-22T17:11:18.871034Z info klog Content-Type: application/vnd.kubernetes.protobuf
2024-02-22T17:11:18.871070Z info klog X-Kubernetes-Pf-Flowschema-Uid: 48115a1f-2c46-4eb8-8350-bab2930df0cf
2024-02-22T17:11:18.871084Z info klog X-Kubernetes-Pf-Prioritylevel-Uid: 329063b9-b080-4896-962f-c405f977c3d0
2024-02-22T17:11:18.871095Z info klog Date: Thu, 22 Feb 2024 17:11:18 GMT
2024-02-22T17:11:18.871120Z info klog Audit-Id: ddfce54e-83f4-4e55-a896-eb900d4df924
2024-02-22T17:11:18.871153Z info klog Cache-Control: no-cache, private
2024-02-22T17:11:18.941864Z debug klog Response Body:
00000000 6b 38 73 00 0a 09 0a 02 76 31 12 03 50 6f 64 12 |k8s.....v1..Pod.|
00000010 93 63 0a 90 23 0a 1a 6d 79 61 70 70 2d 77 65 62 |.c..#..myapp-web|
00000020 2d 36 64 63 39 38 37 63 66 62 38 2d 76 77 35 6e |-6dc987cfb8-vw5n|
00000030 78 12 15 6d 79 61 70 70 2d 77 65 62 2d 36 64 63 |x..myapp-web-6dc|
00000040 39 38 37 63 66 62 38 2d 1a 05 6d 79 61 70 70 22 |987cfb8-..myapp"|
00000050 00 2a 24 63 36 34 63 33 39 62 61 2d 39 30 65 32 |.*$c64c39ba-90e2|
00000060 2d 34 31 64 37 2d 62 30 33 35 2d 66 66 34 64 61 |-41d7-b035-ff4da|
00000070 34 33 32 64 39 64 63 32 08 38 30 37 30 31 38 37 |432d9dc2.8070187|
00000080 30 38 00 42 08 08 8f fe dd ae 06 10 00 5a 10 0a |08.B.........Z..|
00000090 03 61 70 70 12 09 6d 79 61 70 70 2d 77 65 62 5a |.app..myapp-webZ|
000000a0 22 0a 1b 61 70 70 2e 6b 75 62 65 72 6e 65 74 65 |"..app.kubernete|
000000b0 73 2e 69 6f 2f 63 6f 6d 70 6f 6e 65 6e 74 12 03 |s.io/component..|
000000c0 77 65 62 5a 23 0a 1a 61 70 70 2e 6b 75 62 65 72 |webZ#..app.kuber|
000000d0 6e 65 74 65 73 2e 69 6f 2f 69 6e 73 74 61 6e 63 |netes.io/instanc|
000000e0 65 12 05 6d 79 61 70 70 5a 24 0a 1c 61 70 70 2e |e..myappZ$..app.|
000000f0 6b 75 62 65 72 6e 65 74 65 73 2e 69 6f 2f 6d 61 |kubernetes.io/ma|
00000100 6e 61 67 65 64 2d 62 79 12 04 48 65 6c 6d 5a 23 |naged-by..HelmZ#|
00000110 0a 16 61 70 70 2e 6b 75 62 65 72 6e 65 74 65 73 |..app.kubernetes|
00000120 2e 69 6f 2f 6e 61 6d 65 12 09 6d 79 61 70 70 2d |.io/name..myapp-|
00000130 77 65 62 5a 2a 0a 19 61 70 70 2e 6b 75 62 65 72 |webZ*..app.kuber|
00000140 6e 65 74 65 73 2e 69 6f 2f 70 61 72 74 2d 6f 66 |netes.io/part-of|
00000150 12 0d 62 72 61 69 6e 62 61 73 65 6c 69 6e 65 5a |..brainbaselineZ|
00000160 45 0a 19 61 70 70 2e 6b 75 62 65 72 6e 65 74 65 |E..app.kubernete|
00000170 73 2e 69 6f 2f 76 65 72 73 69 6f 6e 12 28 64 30 |s.io/version.(d0|
00000180 34 61 34 34 62 36 37 63 64 66 30 61 37 64 61 30 |4a44b67cdf0a7da0|
00000190 61 65 65 63 64 64 38 63 36 38 30 61 31 65 35 62 |aeecdd8c680a1e5b|
000001a0 38 34 39 35 32 32 5a 25 0a 0d 68 65 6c 6d 2e 73 |849522Z%..helm.s|
000001b0 68 2f 63 68 61 72 74 12 14 62 72 61 69 6e 62 61 |h/chart..brainba|
000001c0 73 65 6c 69 6e 65 2d 30 2e 30 2e 35 39 5a 1f 0a |seline-0.0.59Z..|
000001d0 11 70 6f 64 2d 74 65 6d 70 6c 61 74 65 2d 68 61 |.pod-template-ha|
000001e0 73 68 12 0a 36 64 63 39 38 37 63 66 62 38 5a 22 |sh..6dc987cfb8Z"|
000001f0 0a 19 73 65 63 75 72 69 74 79 2e 69 73 74 69 6f |..security.istio|
00000200 2e 69 6f 2f 74 6c 73 4d 6f 64 65 12 05 69 73 74 |.io/tlsMode..ist|
00000210 69 6f 5a 2c 0a 1f 73 65 72 76 69 63 65 2e 69 73 |ioZ,..service.is|
00000220 74 69 6f 2e 69 6f 2f 63 61 6e 6f 6e 69 63 61 6c |tio.io/canonical|
00000230 2d 6e 61 6d 65 12 09 6d 79 61 70 70 2d 77 65 62 |-name..myapp-web|
00000240 5a 4f 0a 23 73 65 72 76 69 63 65 2e 69 73 74 69 |ZO.#service.isti|
00000250 6f 2e 69 6f 2f 63 61 6e 6f 6e 69 63 61 6c 2d 72 |o.io/canonical-r|
00000260 65 76 69 73 69 6f 6e 12 28 64 30 34 61 34 34 62 |evision.(d04a44b|
00000270 36 37 63 64 66 30 61 37 64 61 30 61 65 65 63 64 |67cdf0a7da0aeecd|
00000280 64 38 63 36 38 30 61 31 65 35 62 38 34 39 35 32 |d8c680a1e5b84952|
00000290 32 5a 33 0a 07 76 65 72 73 69 6f 6e 12 28 64 30 |2Z3..version.(d0|
000002a0 34 61 34 34 62 36 37 63 64 66 30 61 37 64 61 30 |4a44b67cdf0a7da0|
000002b0 61 65 65 63 64 64 38 63 36 38 30 61 31 65 35 62 |aeecdd8c680a1e5b|
000002c0 38 34 39 35 32 32 62 17 0a 0c 69 73 74 69 6f 2e |849522b...istio.|
000002d0 69 6f 2f 72 65 76 12 07 64 65 66 61 75 6c 74 62 |io/rev..defaultb|
000002e0 2e 0a 27 6b 75 62 65 63 74 6c 2e 6b 75 62 65 72 |..'kubectl.kuber|
000002f0 6e 65 74 65 73 2e 69 6f 2f 64 65 66 61 75 6c 74 |netes.io/default|
00000300 2d 63 6f 6e 74 61 69 6e 65 72 12 03 77 65 62 62 |-container..webb|
00000310 33 0a 2c 6b 75 62 65 63 74 6c 2e 6b 75 62 65 72 |3.,kubectl.kuber|
00000320 6e 65 74 65 73 2e 69 6f 2f 64 65 66 61 75 6c 74 |netes.io/default|
00000330 2d 6c 6f 67 73 2d 63 6f 6e 74 61 69 6e 65 72 12 |-logs-container.|
00000340 03 77 65 62 62 27 0a 12 70 72 6f 6d 65 74 68 65 |.webb'..promethe|
00000350 75 73 2e 69 6f 2f 70 61 74 68 12 11 2f 73 74 61 |us.io/path../sta|
00000360 74 73 2f 70 72 6f 6d 65 74 68 65 75 73 62 1b 0a |ts/prometheusb..|
00000370 12 70 72 6f 6d 65 74 68 65 75 73 2e 69 6f 2f 70 |.prometheus.io/p|
00000380 6f 72 74 12 05 31 35 30 32 30 62 1c 0a 14 70 72 |ort..15020b...pr|
00000390 6f 6d 65 74 68 65 75 73 2e 69 6f 2f 73 63 72 61 |ometheus.io/scra|
000003a0 70 65 12 04 74 72 75 65 62 5e 0a 15 70 72 6f 78 |pe..trueb^..prox|
000003b0 79 2e 69 73 74 69 6f 2e 69 6f 2f 63 6f 6e 66 69 |y.istio.io/confi|
000003c0 67 12 45 74 65 72 6d 69 6e 61 74 69 6f 6e 44 72 |g.EterminationDr|
000003d0 61 69 6e 44 75 72 61 74 69 6f 6e 3a 20 33 30 73 |ainDuration: 30s|
000003e0 0a 68 6f 6c 64 41 70 70 6c 69 63 61 74 69 6f 6e |.holdApplication|
000003f0 55 6e 74 69 6c 50 72 6f 78 79 53 74 61 72 74 73 |UntilProxyStarts|
00000400 3a 20 66 61 6c 73 65 0a 62 97 02 0a 17 73 69 64 |: false.b....sid|
00000410 65 63 61 72 2e 69 73 74 69 6f 2e 69 6f 2f 73 74 |ecar.istio.io/st|
00000420 61 74 75 73 12 fb 01 7b 22 69 6e 69 74 43 6f 6e |atus...{"initCon|
00000430 74 61 69 6e 65 72 73 22 3a 5b 22 69 73 74 69 6f |tainers":["istio|
00000440 2d 69 6e 69 74 22 2c 22 69 73 74 69 6f 2d 70 72 |-init","istio-pr|
00000450 6f 78 79 22 5d 2c 22 63 6f 6e 74 61 69 6e 65 72 |oxy"],"container|
00000460 73 22 3a 6e 75 6c 6c 2c 22 76 6f 6c 75 6d 65 73 |s":null,"volumes|
00000470 22 3a 5b 22 77 6f 72 6b 6c 6f 61 64 2d 73 6f 63 |":["workload-soc|
00000480 6b 65 74 22 2c 22 63 72 65 64 65 6e 74 69 61 6c |ket","credential|
00000490 2d 73 6f 63 6b 65 74 22 2c 22 77 6f 72 6b 6c 6f |-socket","worklo|
000004a0 61 64 2d 63 65 72 74 73 22 2c 22 69 73 74 69 6f |ad-certs","istio|
000004b0 2d 65 6e 76 6f 79 22 2c 22 69 73 74 69 6f 2d 64 |-envoy","istio-d|
000004c0 61 74 61 22 2c 22 69 73 74 69 6f 2d 70 6f 64 69 |ata","istio-podi|
000004d0 6e 66 6f 22 2c 22 69 73 74 69 6f 2d 74 6f 6b 65 |nfo","istio-toke|
000004e0 6e 22 2c 22 69 73 74 69 6f 64 2d 63 61 2d 63 65 |n","istiod-ca-ce|
000004f0 72 74 22 5d 2c 22 69 6d 61 67 65 50 75 6c 6c 53 |rt"],"imagePullS|
00000500 65 63 72 65 74 73 22 3a 6e 75 6c 6c 2c 22 72 65 |ecrets":null,"re|
00000510 76 69 73 69 6f 6e 22 3a 22 64 65 66 61 75 6c 74 |vision":"default|
00000520 22 7d 6a 55 0a 0a 52 65 70 6c 69 63 61 53 65 74 |"}jU..ReplicaSet|
00000530 1a 14 6d 79 61 70 70 2d 77 65 62 2d 36 64 63 39 |..myapp-web-6dc9|
00000540 38 37 63 66 62 38 22 24 61 38 61 64 35 64 34 33 |87cfb8"$a8ad5d43|
00000550 2d 65 38 37 39 2d 34 63 39 32 2d 62 31 38 61 2d |-e879-4c92-b18a-|
00000560 64 64 32 30 38 64 33 33 30 34 64 61 2a 07 61 70 |dd208d3304da*.ap|
00000570 70 73 2f 76 31 30 01 38 01 8a 01 ae 12 0a 17 6b |ps/v10.8.......k|
00000580 75 62 65 2d 63 6f 6e 74 72 6f 6c 6c 65 72 2d 6d |ube-controller-m|
00000590 61 6e 61 67 65 72 12 06 55 70 64 61 74 65 1a 02 |anager..Update..|
000005a0 76 31 22 08 08 8f fe dd ae 06 10 00 32 08 46 69 |v1".........2.Fi|
000005b0 65 6c 64 73 56 31 3a f0 11 0a ed 11 7b 22 66 3a |eldsV1:.....{"f:|
000005c0 6d 65 74 61 64 61 74 61 22 3a 7b 22 66 3a 61 6e |metadata":{"f:an|
000005d0 6e 6f 74 61 74 69 6f 6e 73 22 3a 7b 22 2e 22 3a |notations":{".":|
000005e0 7b 7d 2c 22 66 3a 70 72 6f 78 79 2e 69 73 74 69 |{},"f:proxy.isti|
000005f0 6f 2e 69 6f 2f 63 6f 6e 66 69 67 22 3a 7b 7d 7d |o.io/config":{}}|
00000600 2c 22 66 3a 67 65 6e 65 72 61 74 65 4e 61 6d 65 |,"f:generateName|
00000610 22 3a 7b 7d 2c 22 66 3a 6c 61 62 65 6c 73 22 3a |":{},"f:labels":|
00000620 7b 22 2e 22 3a 7b 7d 2c 22 66 3a 61 70 70 22 3a |{".":{},"f:app":|
00000630 7b 7d 2c 22 66 3a 61 70 70 2e 6b 75 62 65 72 6e |{},"f:app.kubern|
00000640 65 74 65 73 2e 69 6f 2f 63 6f 6d 70 6f 6e 65 6e |etes.io/componen|
00000650 74 22 3a 7b 7d 2c 22 66 3a 61 70 70 2e 6b 75 62 |t":{},"f:app.kub|
00000660 65 72 6e 65 74 65 73 2e 69 6f 2f 69 6e 73 74 61 |ernetes.io/insta|
00000670 6e 63 65 22 3a 7b 7d 2c 22 66 3a 61 70 70 2e 6b |nce":{},"f:app.k|
00000680 75 62 65 72 6e 65 74 65 73 2e 69 6f 2f 6d 61 6e |ubernetes.io/man|
00000690 61 67 65 64 2d 62 79 22 3a 7b 7d 2c 22 66 3a 61 |aged-by":{},"f:a|
000006a0 70 70 2e 6b 75 62 65 72 6e 65 74 65 73 2e 69 6f |pp.kubernetes.io|
000006b0 2f 6e 61 6d 65 22 3a 7b 7d 2c 22 66 3a 61 70 70 |/name":{},"f:app|
000006c0 2e 6b 75 62 65 72 6e 65 74 65 73 2e 69 6f 2f 70 |.kubernetes.io/p|
000006d0 61 72 74 2d 6f 66 22 3a 7b 7d 2c 22 66 3a 61 70 |art-of":{},"f:ap|
000006e0 70 2e 6b 75 62 65 72 6e 65 74 65 73 2e 69 6f 2f |p.kubernetes.io/|
000006f0 76 65 72 73 69 6f 6e 22 3a 7b 7d 2c 22 66 3a 68 |version":{},"f:h|
00000700 65 6c 6d 2e 73 68 2f 63 68 61 72 74 22 3a 7b 7d |elm.sh/chart":{}|
00000710 2c 22 66 3a 70 6f 64 2d 74 65 6d 70 6c 61 74 65 |,"f:pod-template|
00000720 2d 68 61 73 68 22 3a 7b 7d 2c 22 66 3a 76 65 72 |-hash":{},"f:ver|
00000730 73 69 6f 6e 22 3a 7b 7d 7d 2c 22 66 3a 6f 77 6e |sion":{}},"f:own|
00000740 65 72 52 65 66 65 72 65 6e 63 65 73 22 3a 7b 22 |erReferences":{"|
00000750 2e 22 3a 7b 7d 2c 22 6b 3a 7b 5c 22 75 69 64 5c |.":{},"k:{\"uid\|
00000760 22 3a 5c 22 61 38 61 64 35 64 34 33 2d 65 38 37 |":\"a8ad5d43-e87|
00000770 39 2d 34 63 39 32 2d 62 31 38 61 2d 64 64 32 30 |9-4c92-b18a-dd20|
00000780 38 64 33 33 30 34 64 61 5c 22 7d 22 3a 7b 7d 7d |8d3304da\"}":{}}|
00000790 7d 2c 22 66 3a 73 70 65 63 22 3a 7b 22 66 3a 63 |},"f:spec":{"f:c|
000007a0 6f 6e 74 61 69 6e 65 72 73 22 3a 7b 22 6b 3a 7b |ontainers":{"k:{|
000007b0 5c 22 6e 61 6d 65 5c 22 3a 5c 22 77 65 62 5c 22 |\"name\":\"web\"|
000007c0 7d 22 3a 7b 22 2e 22 3a 7b 7d 2c 22 66 3a 61 72 |}":{".":{},"f:ar|
000007d0 67 73 22 3a 7b 7d 2c 22 66 3a 65 6e 76 46 72 6f |gs":{},"f:envFro|
000007e0 6d 22 3a 7b 7d 2c 22 66 3a 69 6d 61 67 65 22 3a |m":{},"f:image":|
000007f0 7b 7d 2c 22 66 3a 69 6d 61 67 65 50 75 6c 6c 50 |{},"f:imagePullP|
00000800 6f 6c 69 63 79 22 3a 7b 7d 2c 22 66 3a 6c 69 76 |olicy":{},"f:liv|
00000810 65 6e 65 73 73 50 72 6f 62 65 22 3a 7b [truncated 52558 chars]
2024-02-22T17:11:18.947794Z info klog curl -v -XPOST -H "User-Agent: istioctl/1.20.3" -H "X-Stream-Protocol-Version: portforward.k8s.io" 'https://REDACTED.gr7.us-east-2.eks.amazonaws.com/api/v1/namespaces/myapp/pods/myapp-web-6dc987cfb8-vw5nx/portforward'
2024-02-22T17:11:18.948992Z info klog HTTP Trace: DNS Lookup for REDACTED.gr7.us-east-2.eks.amazonaws.com resolved to [{10.34.7.111 } {10.34.105.167 }]
2024-02-22T17:11:19.048088Z info klog HTTP Trace: Dial to tcp:10.34.7.111:443 succeed
2024-02-22T17:11:19.300318Z info klog POST https://REDACTED.gr7.us-east-2.eks.amazonaws.com/api/v1/namespaces/myapp/pods/myapp-web-6dc987cfb8-vw5nx/portforward 101 Switching Protocols in 352 milliseconds
2024-02-22T17:11:19.300425Z info klog HTTP Statistics: DNSLookup 1 ms Dial 99 ms TLSHandshake 0 ms Duration 352 ms
2024-02-22T17:11:19.300454Z info klog Response Headers:
2024-02-22T17:11:19.300490Z info klog Upgrade: SPDY/3.1
2024-02-22T17:11:19.300516Z info klog X-Stream-Protocol-Version: portforward.k8s.io
2024-02-22T17:11:19.300539Z info klog Connection: Upgrade
2024-02-22T17:11:19.563738Z error klog an error occurred forwarding 60301 -> 15000: error forwarding port 15000 to pod 37e6dae5a6d46df325420609ffd9cfddef0ef20377ce69082961f34578d4bf5f, uid : failed to execute portforward in network namespace "/var/run/netns/cni-ef0013a0-a3fd-d141-8058-905f556ca236": failed to connect to localhost:15000 inside namespace "37e6dae5a6d46df325420609ffd9cfddef0ef20377ce69082961f34578d4bf5f", IPv4: dial tcp4 127.0.0.1:15000: connect: connection refused IPv6 dial tcp6: address localhost: no suitable address found
2024-02-22T17:11:19.564620Z error port forward failed: lost connection to pod
Error: failed to execute command on myapp-web-6dc987cfb8-vw5nx.myapp sidecar: failure running port forward process: Get "http://localhost:60301/config_dump": EOF
/etc/hosts:
app@myapp-web-5f5b6c695f-n97jz:~$ cat /etc/hosts
# Kubernetes-managed hosts file.
127.0.0.1 localhost
::1 localhost ip6-localhost ip6-loopback
fe00::0 ip6-localnet
fe00::0 ip6-mcastprefix
fe00::1 ip6-allnodes
fe00::2 ip6-allrouters
2600:1f16:40:5306:1f57::6 myapp-web-5f5b6c695f-n97jz
cc @codeverifier
Version
Istio 1.20.3
AWS EKS 1.29 IPv6-enabled
Additional Information
I did find https://github.com/istio/istio/issues/34358 but EKS is currently running containerd 1.7.11 and I added the /etc/hosts above which appears to already have localhost properly mapped.
It looks like the port forwarder is using the IPv4 address. I don't have an IPv6 cluster, so cc @zhlsunshine to take a look
Hi @mmerickel & @hanxiaop, it's obvious that the error is caused by the port forwarder is using IPv4 address to dial remote IPv6 address according to above description in your environment. I can confirm that the port forwarder in istioctl would use localhost when executing istioctl pc in your case. So I also believe that something happened during localhost to 127.0.0.1 and to dial remote IPv6 address.
So @mmerickel can you try to change the /etc/hosts to make ::1 localhost be the first entry? Or is it possible to disable AWS EKS 1.29 IPv6 with their weird IPv4 link-local NAT egress?
It’s not possible to turn off the ipv4 NAT in EKS. I suspect you might want to leverage the ISTIO_ENABLE_IPV4_OUTBOUND_LISTENER_FOR_IPV6_CLUSTERS setting that is related to this situation in some way? I will try editing the host to confirm your theory when I get back to my computer.
Try this sed -i 's/::1 localhost6 localhost6.localdomain6/::1 localhost localhost6 localhost6.localdomain6/' /etc/hosts
Make sure you also modify the underlying host / node /etc/hosts.
The way I do it is to run the above command in a pre-boostrap (in TF) step so the node is provisioned with it.
I did things in reverse order - I wasn't able to modify /etc/hosts in the container itself because our apps run as an unprivileged user and I'm not smart enough to circumvent that atm. So instead I modified /etc/hosts on the host itself first - commented out the 127.0.0.1 line and then istioctl pc l myapp-web-5576c48d65-dj5lj worked just fine for the first time. So it seems more related to the host than the container.
Maybe istio could use localhost6 :-)
I'm mostly at the point here where I've put so much time into trying an IPv6 cluster but I think I just have to call it a failure. I know it's not specifically IPv6 and more related to AWS EKS linklocal shenanigans but at the end of the day I do want my hosts to have this linklocal IPv4 NAT because quite simply there's tons of services I need to connect to that are not IPv6 yet and this is the easiest way to do it. Otherwise I'd have to setup DNS64 etc and deal with larger networking nastiness outside of the cluster itself. It's just too messy and unstable right now. :-(
FWIW here is the /etc/hosts file on the EKS managed node itself:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost6 localhost6.localdomain6
I confirmed that using your sed script to add localhost to the ::1 entry does fix the issue as well.
My 2 cents is that istio should know that this is an IPv6 cluster and not rely on localhost for DNS. It should know what interface to hit to get a response and hit that. I don't see this being an EKS issue.
From the deep dive I did a year or so back this was not related to Istio. This is how port-forwarding works in the CRI implementation aka containerd. From what i recall it uses localhost. At the end of the day Istio relies on kube port-forwarding for some of the istioctl commands.
I know that port fowarding is not generally broken. It works for things like istioctl dashboard kiali as well as generic kubectl port-forward commands. The issue is something specific to what istioctl proxy-config is doing.
When you mean generic port-forward is that to the proxy admin port ?
Other services may not be truly listening for tcp6 requests
I know that port fowarding is not generally broken. It works for things like
istioctl dashboard kialias well as generickubectl port-forwardcommands. The issue is something specific to whatistioctl proxy-configis doing.
FWIW dashbord and proxy-config share the same port-forward logic in this case, may need to investigate why only proxy-config command is broken.
When you mean generic port-forward is that to the proxy admin port ?
I just mean I've used kubectl port-forward successfully to various services on an EKS-IPv6 cluster for some apps, not to the proxy admin port specifically. I've only encountered issues with proxy-config. That being said I deleted my IPv6 cluster out of frustration from all of these issues for the short term so can't give you a ton of details right now.
I can confirm btw that I see the same errors if I hit the proxy myself with kubectl port-forward:
kubectl port-forward -n istio-ingressgateway-private istio-ingressgateway-78ddddf548-nlf58 15000:15000
Forwarding from 127.0.0.1:15000 -> 15000
Forwarding from [::1]:15000 -> 15000
Handling connection for 15000
E0305 15:32:56.165931 80794 portforward.go:409] an error occurred forwarding 15000 -> 15000: error forwarding port 15000 to pod 2ff00ec557ed84d8928988130bae68a3a38ef77d0f27908db9b8acf771542875, uid : failed to execute portforward in network namespace "/var/run/netns/cni-a56faa1d-177f-2320-d9a4-a839886fdd7b": failed to connect to localhost:15000 inside namespace "2ff00ec557ed84d8928988130bae68a3a38ef77d0f27908db9b8acf771542875", IPv4: dial tcp4 127.0.0.1:15000: connect: connection refused IPv6 dial tcp6: address localhost: no suitable address found
error: lost connection to pod
If I try with --address ::1 then I get the same error.
Note that in both cases, things startup fine. The errors only appear after issuing a request over the port, for example curl localhost:15000/config_dump.
This is because of the way the proxy binds on tcp6:15000. Currently "ipv4_compat" isnt used anywhere. i believe it first tries to port-forward ipv4 and ipv6.
Again though, it's specific to the istio proxy and not the entire host. I can port-forward other pods right, and obviously istioctl dashboard kiali works as examples. So is there some option I can configure in istio to get this to work? Are istio folks working with EKS folks on fixing it? What's the path forward here?
To change the compatibility of the proxies is a big piece of work, one that I'm sure wel never get to in short/medium term.
Also not sure how all of this behaves with Ambient, particularly the waypoints.
But thank you for raising this with AWS folks as well.
Also not sure how all of this behaves with Ambient, particularly the waypoints.
I think the waypoints will have the same behaviors as sidecars for this particular command.