TLS End to End Issue with Backend Health - Unhealthy, Resulting in 502 Errors

[adamcarter81]

trafficstars

Describe the bug I am trying to configure TLS E2E. The frontend listener, backend pool, health probes all create successfully. When trying to access the URL I receive a 502 from the AppGW The HTTPS health probe if I edit it and click Test - is successful If I go to Backend Health, it shows - Cannot connect to server. Check whether any NSG/UDR/Firewall is blocking access to server. Connection Troubleshoot, manually specify IP of backend on port tcp/443 is succesful

To Reproduce I am using the exported root as the trusted cert on the gateway and it is selected Ok in the front end

I can forward a port via kubectl and access the pod just perfect over https by manipulating the domain to localhost

The cert is a wildcard

The 502 returned in the browser is receiving the correct cert and is valid

It seems the issue is with AGIC -> pod ?

Ingress Controller details

• Output of `kubectl describe pod:

`PS C:\repos\aks-lab-001> kubectl get pod NAME READY STATUS RESTARTS AGE ingress-azure-1590078545-7bf89c5df5-lhh84 2/2 Running 16 3h29m mic-5bf56d5658-6jqbs 1/1 Running 0 9h mic-5bf56d5658-kcgt7 1/1 Running 0 9h nmi-c26ph 1/1 Running 2 85d nmi-wbm7c 1/1 Running 5 85d PS C:\repos\aks-lab-001> kubectl describe pod ingress-azure-1590078545-7bf89c5df5-lhh84 Name: ingress-azure-1590078545-7bf89c5df5-lhh84 Namespace: default Priority: 0 Node: aks-linux-35064155-vmss000000/15.0.0.4 Start Time: Thu, 21 May 2020 17:29:23 +0100 Labels: aadpodidbinding=ingress-azure-1590078545 app=ingress-azure pod-template-hash=7bf89c5df5 release=ingress-azure-1590078545 security.istio.io/tlsMode=istio service.istio.io/canonical-name=ingress-azure service.istio.io/canonical-revision=latest Annotations: prometheus.io/port: 8123 prometheus.io/scrape: true sidecar.istio.io/status: {"version":"fca84600f9d5ec316cf1cf577da902f38bac258ab0fd595ee208ec0203dc0c6d","initContainers":["istio-init"],"containers":["istio-proxy"]... Status: Running IP: 15.0.0.7 IPs: Controlled By: ReplicaSet/ingress-azure-1590078545-7bf89c5df5 Init Containers: istio-init: Container ID: docker://caf5dc3a17278ede6d4ecf28af00b02a7ca632610c7b3ad37926de4fa8b4c4fa Image: docker.io/istio/proxyv2:1.5.4 Image ID: docker-pullable://istio/proxyv2@sha256:e16e2801b7fd93154e8fcb5f4e2fb1240d73349d425b8be90691d48e8b9bb944 Port: Host Port: Command: istio-iptables -p 15001 -z 15006 -u 1337 -m REDIRECT -i * -x

  -b
  *
  -d
  15090,15020
State:          Terminated
  Reason:       Completed
  Exit Code:    0
  Started:      Thu, 21 May 2020 17:29:25 +0100
  Finished:     Thu, 21 May 2020 17:29:25 +0100
Ready:          True
Restart Count:  0
Limits:
  cpu:     100m
  memory:  50Mi
Requests:
  cpu:        10m
  memory:     10Mi
Environment:  <none>
Mounts:       <none>

Containers: ingress-azure: Container ID: docker://85776a692fef00987f2f2ec4b3c5977cd6ee976e6700c2e3ef1aabc2329141c8 Image: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.0.0 Image ID: docker-pullable://mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:c295f99ae66443c5a392fd894620fcd1fc313b9efdec96d13f166fefb29780a9 Port: Host Port: State: Running Started: Thu, 21 May 2020 18:47:08 +0100 Last State: Terminated Reason: Error Exit Code: 2 Started: Thu, 21 May 2020 18:41:37 +0100 Finished: Thu, 21 May 2020 18:41:58 +0100 Ready: True Restart Count: 16 Liveness: http-get http://:15020/app-health/ingress-azure/livez delay=15s timeout=1s period=20s #success=1 #failure=3 Readiness: http-get http://:15020/app-health/ingress-azure/readyz delay=5s timeout=1s period=10s #success=1 #failure=3 Environment Variables from: ingress-azure-1590078545 ConfigMap Optional: false Environment: AZURE_CONTEXT_LOCATION: /etc/appgw/azure.json AGIC_POD_NAME: ingress-azure-1590078545-7bf89c5df5-lhh84 (v1:metadata.name) AGIC_POD_NAMESPACE: default (v1:metadata.namespace) Mounts: /etc/appgw/azure.json from azure (rw) /var/run/secrets/kubernetes.io/serviceaccount from ingress-azure-1590078545-token-v44t6 (ro) istio-proxy: Container ID: docker://66e8eb4b7e86fc7418c75d8deb910adce4671cd9b68eebce6ea615c2b2598f97 Image: docker.io/istio/proxyv2:1.5.4 Image ID: docker-pullable://istio/proxyv2@sha256:e16e2801b7fd93154e8fcb5f4e2fb1240d73349d425b8be90691d48e8b9bb944 Port: 15090/TCP Host Port: 0/TCP Args: proxy sidecar --domain $(POD_NAMESPACE).svc.cluster.local --configPath /etc/istio/proxy --binaryPath /usr/local/bin/envoy --serviceCluster ingress-azure.$(POD_NAMESPACE) --drainDuration 45s --parentShutdownDuration 1m0s --discoveryAddress istiod.istio-system.svc:15012 --zipkinAddress zipkin.istio-system:9411 --proxyLogLevel=warning --proxyComponentLogLevel=misc:error --connectTimeout 10s --proxyAdminPort 15000 --concurrency 2 --controlPlaneAuthPolicy NONE --dnsRefreshRate 300s --statusPort 15020 --trust-domain=cluster.local --controlPlaneBootstrap=false State: Running Started: Thu, 21 May 2020 17:29:27 +0100 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 1Gi Requests: cpu: 100m memory: 128Mi Readiness: http-get http://:15020/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30 Environment: JWT_POLICY: third-party-jwt PILOT_CERT_PROVIDER: istiod CA_ADDR: istio-pilot.istio-system.svc:15012 POD_NAME: ingress-azure-1590078545-7bf89c5df5-lhh84 (v1:metadata.name) POD_NAMESPACE: default (v1:metadata.namespace) INSTANCE_IP: (v1:status.podIP) SERVICE_ACCOUNT: (v1:spec.serviceAccountName) HOST_IP: (v1:status.hostIP) ISTIO_META_POD_PORTS: [ ] ISTIO_META_APP_CONTAINERS: [ ingress-azure ] ISTIO_META_CLUSTER_ID: Kubernetes ISTIO_META_POD_NAME: ingress-azure-1590078545-7bf89c5df5-lhh84 (v1:metadata.name) ISTIO_META_CONFIG_NAMESPACE: default (v1:metadata.namespace) ISTIO_META_INTERCEPTION_MODE: REDIRECT ISTIO_METAJSON_ANNOTATIONS: {"prometheus.io/port":"8123","prometheus.io/scrape":"true"}

  ISTIO_META_WORKLOAD_NAME:      ingress-azure-1590078545
  ISTIO_META_OWNER:              kubernetes://apis/apps/v1/namespaces/default/deployments/ingress-azure-1590078545
  ISTIO_META_MESH_ID:            cluster.local
  ISTIO_KUBE_APP_PROBERS:        {"/app-health/ingress-azure/livez":{"httpGet":{"path":"/health/alive","port":8123,"scheme":"HTTP"},"timeoutSeconds":1},"/app-health/ingress-azure/readyz":{"httpGet":{"path":"/health/ready","port":8123,"scheme":"HTTP"},"timeoutSeconds":1}}
Mounts:
  /etc/istio/pod from podinfo (rw)
  /etc/istio/proxy from istio-envoy (rw)
  /var/run/secrets/istio from istiod-ca-cert (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from ingress-azure-1590078545-token-v44t6 (ro)
  /var/run/secrets/tokens from istio-token (rw)

Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: azure: Type: HostPath (bare host directory volume) Path: /etc/kubernetes/azure.json HostPathType: File ingress-azure-1590078545-token-v44t6: Type: Secret (a volume populated by a Secret) SecretName: ingress-azure-1590078545-token-v44t6 Optional: false istio-envoy: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: Memory SizeLimit: podinfo: Type: DownwardAPI (a volume populated by information about the pod) Items: metadata.labels -> labels metadata.annotations -> annotations istio-token: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 43200 istiod-ca-cert: Type: ConfigMap (a volume populated by a ConfigMap) Name: istio-ca-root-cert Optional: false QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: `

• Output of kubectl logs <ingress controller>: -- App Gwy config --} I0521 19:40:50.369045 1 mutate_app_gateway.go:182] Applied App Gateway config in 20.446695951s I0521 19:40:50.369062 1 mutate_app_gateway.go:198] cache: Updated with latest applied config. I0521 19:40:50.370154 1 mutate_app_gateway.go:203] END AppGateway deployment`

• Any Azure support tickets associated with this issue. N/A

[adamcarter81]

A little more information - my backend is the Istio ingress gateway, and I have AGIC working fine with HTTP to Istio ingress, but I now want to add HTTPS.

I have configured TLS pass through in istio gateway - and if I forward a local port, or hit the istio gateway on the external pod it works fine - I see my test site, indicating that TLS pass through and certificates are configured correctly. The issue seems to be with AGIC to Istio.

I guess I could try going directly from AGIC to the end service, bypassing Istio ingress, but then I would lose the ability to use Istio to observe the mesh in this case. Should this be a workable configuration or has it been confirmed working by anyone?

[3quanfeng]

Hi @adamcarter81 , it looks like your issue happens when appgw talks to istio gateway via TLS. with the version of AGIC you are using(1.0.0), AGIC only support https when Pods are using certificate signed by a well-known CA. AGIC 1.2.0-rc2 supports to configure with your own root certificate with a new annotation

[adamcarter81]

Ah, I thought I was on 1.2.0-rc2, I remember downgrading before collecting the last logs. 1.2.0-rc2 gives me an error of wrong protocol when creating the probe:

I0522 19:24:34.046125       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0522 19:24:34.192093       1 mutate_app_gateway.go:183] END AppGateway deployment
E0522 19:24:34.192237       1 controller.go:141] network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ApplicationGatewayProbeProtocolMustMatchBackendHttpSettinsProtocol" Message="Probe /subscriptions/d3af426f-de5c-48ea-a5e8-f645a6ac85e9/resourceGroups/rg-aks-lab/providers/Microsoft.Network/applicationGateways/ApplicationGateway1/probes/pb-istio-system-istio-ingressgateway-443-agic-ingress-https protocol (Http) does not match Backend Http Setting /subscriptions/d3af426f-de5c-48ea-a5e8-f645a6ac85e9/resourceGroups/rg-aks-lab/providers/Microsoft.Network/applicationGateways/ApplicationGateway1/backendHttpSettingsCollection/bp-istio-system-istio-ingressgateway-443-443-agic-ingress-https protocol (Https)." Details=[]
E0522 19:24:34.192252       1 worker.go:62] Error processing event.network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=400 -- Original Error: Code="ApplicationGatewayProbeProtocolMustMatchBackendHttpSettinsProtocol" Message="Probe /subscriptions/d3af426f-de5c-48ea-a5e8-f645a6ac85e9/resourceGroups/rg-aks-lab/providers/Microsoft.Network/applicationGateways/ApplicationGateway1/probes/pb-istio-system-istio-ingressgateway-443-agic-ingress-https protocol (Http) does not match Backend Http Setting /subscriptions/d3af426f-de5c-48ea-a5e8-f645a6ac85e9/resourceGroups/rg-aks-lab/providers/Microsoft.Network/applicationGateways/ApplicationGateway1/backendHttpSettingsCollection/bp-istio-system-istio-ingressgateway-443-443-agic-ingress-https protocol (Https)." Details=[]

I thought this was addressed in 851 but I never got to testing it properly until now

I have the root cert specified with the secret - I can see in the logs it picks this up when building the config OK.

deployment yaml: `apiVersion: extensions/v1beta1 kind: Ingress metadata:

name: agic-ingress-https annotations: kubernetes.io/ingress.class: azure/application-gateway appgw.ingress.kubernetes.io/backend-hostname: "appgw.domain.com" appgw.ingress.kubernetes.io/cookie-based-affinity: "true" appgw.ingress.kubernetes.io/backend-protocol: "Https" appgw.ingress.kubernetes.io/appgw-trusted-root-certificate: wildcard-domain-com-root spec: tls: - hosts: - test.domain.com secretName: wildcard-testdomain-com

rules:

• host: test.domain.com http: paths:
  • backend: #serviceName: svc-agic-ingress-https serviceName: istio-ingressgateway servicePort: 443 path: /*`

The probes it tries to create - you can see the last one is the problem one with trying to specify http as the protocol for the 443 setting:

-- App Gwy config -- "probes": [ -- App Gwy config -- { -- App Gwy config -- "id": "/subscriptions/d3af426f-de5c-48ea-a5e8-f645a6ac85e9/resourceGroups/rg-aks-lab/providers/Microsoft.Network/applicationGateways/ApplicationGateway1/probes/defaultprobe-Http", -- App Gwy config -- "name": "defaultprobe-Http", -- App Gwy config -- "properties": { -- App Gwy config -- "host": "localhost", -- App Gwy config -- "interval": 30, -- App Gwy config -- "match": {}, -- App Gwy config -- "minServers": 0, -- App Gwy config -- "path": "/", -- App Gwy config -- "pickHostNameFromBackendHttpSettings": false, -- App Gwy config -- "protocol": "Http", -- App Gwy config -- "timeout": 30, -- App Gwy config -- "unhealthyThreshold": 3 -- App Gwy config -- } -- App Gwy config -- }, -- App Gwy config -- { -- App Gwy config -- "id": "/subscriptions/d3af426f-de5c-48ea-a5e8-f645a6ac85e9/resourceGroups/rg-aks-lab/providers/Microsoft.Network/applicationGateways/ApplicationGateway1/probes/defaultprobe-Https", -- App Gwy config -- "name": "defaultprobe-Https", -- App Gwy config -- "properties": { -- App Gwy config -- "host": "localhost", -- App Gwy config -- "interval": 30, -- App Gwy config -- "match": {}, -- App Gwy config -- "minServers": 0, -- App Gwy config -- "path": "/", -- App Gwy config -- "pickHostNameFromBackendHttpSettings": false, -- App Gwy config -- "protocol": "Https", -- App Gwy config -- "timeout": 30, -- App Gwy config -- "unhealthyThreshold": 3 -- App Gwy config -- } -- App Gwy config -- }, -- App Gwy config -- { -- App Gwy config -- "id": "/subscriptions/d3af426f-de5c-48ea-a5e8-f645a6ac85e9/resourceGroups/rg-aks-lab/providers/Microsoft.Network/applicationGateways/ApplicationGateway1/probes/pb-istio-system-istio-ingressgateway-443-agic-ingress-https", -- App Gwy config -- "name": "pb-istio-system-istio-ingressgateway-443-agic-ingress-https", -- App Gwy config -- "properties": { -- App Gwy config -- "host": "appgw.domain.com", -- App Gwy config -- "interval": 2, -- App Gwy config -- "match": {}, -- App Gwy config -- "minServers": 0, -- App Gwy config -- "path": "/healthz/ready", -- App Gwy config -- "pickHostNameFromBackendHttpSettings": false, -- App Gwy config -- "port": 15020, -- App Gwy config -- "protocol": "Http", -- App Gwy config -- "timeout": 1, -- App Gwy config -- "unhealthyThreshold": 20 -- App Gwy config -- } -- App Gwy config -- } -- App Gwy config -- ],

[3quanfeng]

Hi @adamcarter81 , could you please help describe the pod? thanks!

[adamcarter81]

Hi @3quanfeng here it is: `PS C:\repos\aks-lab-001> kubectl describe pod ingress-azure-1590078545-5f44f5b644-6mkh4 Name: ingress-azure-1590078545-5f44f5b644-6mkh4 Namespace: default Priority: 0 Node: aks-linux-35064155-vmss000000/15.0.0.4 Start Time: Fri, 22 May 2020 20:13:30 +0100 Labels: aadpodidbinding=ingress-azure-1590078545 app=ingress-azure pod-template-hash=5f44f5b644 release=ingress-azure-1590078545 security.istio.io/tlsMode=istio service.istio.io/canonical-name=ingress-azure service.istio.io/canonical-revision=latest Annotations: checksum/config: aca019ed1166faea192c627e0b9562da1292f62392d54327dadcb4e7de23bc28 prometheus.io/port: 8123 prometheus.io/scrape: true sidecar.istio.io/status: {"version":"fca84600f9d5ec316cf1cf577da902f38bac258ab0fd595ee208ec0203dc0c6d","initContainers":["istio-init"],"containers":["istio-proxy"]... Status: Running IP: 15.0.0.11 IPs: Controlled By: ReplicaSet/ingress-azure-1590078545-5f44f5b644 Init Containers: istio-init: Container ID: docker://3fb2b030e09a7593a2dceaba2e060c8aab83e102a17857b1c3a53706401e1e46 Image: docker.io/istio/proxyv2:1.5.4 Image ID: docker-pullable://istio/proxyv2@sha256:e16e2801b7fd93154e8fcb5f4e2fb1240d73349d425b8be90691d48e8b9bb944 Port: Host Port: Command: istio-iptables -p 15001 -z 15006 -u 1337 -m REDIRECT -i * -x

  -b
  *
  -d
  15090,15020
State:          Terminated
  Reason:       Completed
  Exit Code:    0
  Started:      Fri, 22 May 2020 20:13:32 +0100
  Finished:     Fri, 22 May 2020 20:13:32 +0100
Ready:          True
Restart Count:  0
Limits:
  cpu:     100m
  memory:  50Mi
Requests:
  cpu:        10m
  memory:     10Mi
Environment:  <none>
Mounts:       <none>

Containers: ingress-azure: Container ID: docker://48e1609b4ad46168d99ecf46bfe52be2003c4738b6b1ce36d038f121731c274c Image: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.2.0-rc2 Image ID: docker-pullable://mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:c1ba4ef96f7d6b31f4c7a4e524253b50480c4933e414f4b2763e13be6b5c29de Port: Host Port: State: Running Started: Fri, 22 May 2020 20:13:33 +0100 Ready: True Restart Count: 0 Liveness: http-get http://:15020/app-health/ingress-azure/livez delay=15s timeout=1s period=20s #success=1 #failure=3 Readiness: http-get http://:15020/app-health/ingress-azure/readyz delay=5s timeout=1s period=10s #success=1 #failure=3 Environment Variables from: ingress-azure-1590078545 ConfigMap Optional: false Environment: AZURE_CLOUD_PROVIDER_LOCATION: /etc/appgw/azure.json AGIC_POD_NAME: ingress-azure-1590078545-5f44f5b644-6mkh4 (v1:metadata.name) AGIC_POD_NAMESPACE: default (v1:metadata.namespace) Mounts: /etc/appgw/azure.json from azure (rw) /var/run/secrets/kubernetes.io/serviceaccount from ingress-azure-1590078545-token-88kdt (ro) istio-proxy: Container ID: docker://f9faf0727022e5e6bffb814a8a45cad92802678aac31feeeba724ed33d1ab05c Image: docker.io/istio/proxyv2:1.5.4 Image ID: docker-pullable://istio/proxyv2@sha256:e16e2801b7fd93154e8fcb5f4e2fb1240d73349d425b8be90691d48e8b9bb944 Port: 15090/TCP Host Port: 0/TCP Args: proxy sidecar --domain $(POD_NAMESPACE).svc.cluster.local --configPath /etc/istio/proxy --binaryPath /usr/local/bin/envoy --serviceCluster ingress-azure.$(POD_NAMESPACE) --drainDuration 45s --parentShutdownDuration 1m0s --discoveryAddress istiod.istio-system.svc:15012 --zipkinAddress zipkin.istio-system:9411 --proxyLogLevel=warning --proxyComponentLogLevel=misc:error --connectTimeout 10s --proxyAdminPort 15000 --concurrency 2 --controlPlaneAuthPolicy NONE --dnsRefreshRate 300s --statusPort 15020 --trust-domain=cluster.local --controlPlaneBootstrap=false State: Running Started: Fri, 22 May 2020 20:13:34 +0100 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 1Gi Requests: cpu: 100m memory: 128Mi Readiness: http-get http://:15020/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30 Environment: JWT_POLICY: third-party-jwt PILOT_CERT_PROVIDER: istiod CA_ADDR: istio-pilot.istio-system.svc:15012 POD_NAME: ingress-azure-1590078545-5f44f5b644-6mkh4 (v1:metadata.name) POD_NAMESPACE: default (v1:metadata.namespace) INSTANCE_IP: (v1:status.podIP) SERVICE_ACCOUNT: (v1:spec.serviceAccountName) HOST_IP: (v1:status.hostIP) ISTIO_META_POD_PORTS: [ ] ISTIO_META_APP_CONTAINERS: [ ingress-azure ] ISTIO_META_CLUSTER_ID: Kubernetes ISTIO_META_POD_NAME: ingress-azure-1590078545-5f44f5b644-6mkh4 (v1:metadata.name) ISTIO_META_CONFIG_NAMESPACE: default (v1:metadata.namespace) ISTIO_META_INTERCEPTION_MODE: REDIRECT ISTIO_METAJSON_ANNOTATIONS: {"checksum/config":"aca019ed1166faea192c627e0b9562da1292f62392d54327dadcb4e7de23bc28","prometheus.io/port":"8123","prometheus.io/scrape":"true"}

  ISTIO_META_WORKLOAD_NAME:      ingress-azure-1590078545
  ISTIO_META_OWNER:              kubernetes://apis/apps/v1/namespaces/default/deployments/ingress-azure-1590078545
  ISTIO_META_MESH_ID:            cluster.local
  ISTIO_KUBE_APP_PROBERS:        {"/app-health/ingress-azure/livez":{"httpGet":{"path":"/health/alive","port":8123,"scheme":"HTTP"},"timeoutSeconds":1},"/app-health/ingress-azure/readyz":{"httpGet":{"path":"/health/ready","port":8123,"scheme":"HTTP"},"timeoutSeconds":1}}
Mounts:
  /etc/istio/pod from podinfo (rw)
  /etc/istio/proxy from istio-envoy (rw)
  /var/run/secrets/istio from istiod-ca-cert (rw)
  /var/run/secrets/kubernetes.io/serviceaccount from ingress-azure-1590078545-token-88kdt (ro)
  /var/run/secrets/tokens from istio-token (rw)

Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: azure: Type: HostPath (bare host directory volume) Path: /etc/kubernetes/azure.json HostPathType: File ingress-azure-1590078545-token-88kdt: Type: Secret (a volume populated by a Secret) SecretName: ingress-azure-1590078545-token-88kdt Optional: false istio-envoy: Type: EmptyDir (a temporary directory that shares a pod's lifetime) Medium: Memory SizeLimit: podinfo: Type: DownwardAPI (a volume populated by information about the pod) Items: metadata.labels -> labels metadata.annotations -> annotations istio-token: Type: Projected (a volume that contains injected data from multiple sources) TokenExpirationSeconds: 43200 istiod-ca-cert: Type: ConfigMap (a volume populated by a ConfigMap) Name: istio-ca-root-cert Optional: false QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: `

[3quanfeng]

Thanks a lot @adamabmsft , did you specify scheme in your container readiness probe definition? would you mind to share your template.spec.containers in your k8s deployment?

[adamcarter81]

@3quanfeng Good point, I haven't specified any readiness probe, so it will just use the default. Interesting that 1.0.0 builds this OK, but 1.2.0-rc2 doesn't, so some behaviour has changed there. It is probably best I explicitly configure the probe anyway - I'll give that a try

[adamcarter81]

Thanks a lot @adamabmsft , did you specify scheme in your container readiness probe definition? would you mind to share your template.spec.containers in your k8s deployment?

Here I describe the pod for the istio-ingressgateway. Can see the readiness probe is set to http-get. As the Istio ingressgateway will be required to receive both HTTP and HTTPs traffic, I guess I would need to add the HTTPS scheme so that it has 2 x readiness checks for HTTP & HTTPS. Would AGIC then pick the correct protocol if both are present?

`PS C:\repos\aks-lab-001> kubectl -n istio-system describe pod istio-ingressgateway-bd6f57f65-v6zq7 Name: istio-ingressgateway-bd6f57f65-v6zq7 Namespace: istio-system Priority: 0 Node: aks-linux-35064155-vmss000000/15.0.0.4 Start Time: Wed, 27 May 2020 17:07:31 +0100 Labels: app=istio-ingressgateway chart=gateways heritage=Tiller istio=ingressgateway pod-template-hash=bd6f57f65 release=istio Annotations: sidecar.istio.io/inject: false Status: Running IP: 15.0.0.25 IPs: Controlled By: ReplicaSet/istio-ingressgateway-bd6f57f65 Containers: istio-proxy: Container ID: docker://9e06ca60e166935471d67651030d2d35b309619556b79a085da191bd770fad5f Image: docker.io/istio/proxyv2:1.4.9 Image ID: docker-pullable://istio/proxyv2@sha256:6d14af93c409b1ff5d4ce3a6a3b7eeb902dca879f2e86fbdd97bb16df4589309 Ports: 15020/TCP, 80/TCP, 443/TCP, 15029/TCP, 15030/TCP, 15031/TCP, 15032/TCP, 15443/TCP, 15011/TCP, 8060/TCP, 853/TCP, 15090/TCP Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP, 0/TCP Args: proxy router --domain $(POD_NAMESPACE).svc.cluster.local --proxyLogLevel=warning --proxyComponentLogLevel=misc:error --log_output_level=default:info --drainDuration 45s --parentShutdownDuration 1m0s --connectTimeout 10s --serviceCluster istio-ingressgateway --zipkinAddress zipkin.istio-system:9411 --proxyAdminPort 15000 --statusPort 15020 --controlPlaneAuthPolicy NONE --discoveryAddress istio-pilot.istio-system:15010 --trust-domain=cluster.local State: Running Started: Wed, 27 May 2020 17:07:40 +0100 Ready: True Restart Count: 0 Limits: cpu: 2 memory: 1Gi Requests: cpu: 10m memory: 40Mi Readiness: http-get http://:15020/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30 Environment: NODE_NAME: (v1:spec.nodeName) POD_NAME: istio-ingressgateway-bd6f57f65-v6zq7 (v1:metadata.name) POD_NAMESPACE: istio-system (v1:metadata.namespace) INSTANCE_IP: (v1:status.podIP) HOST_IP: (v1:status.hostIP) SERVICE_ACCOUNT: (v1:spec.serviceAccountName) ISTIO_META_WORKLOAD_NAME: istio-ingressgateway ISTIO_META_OWNER: kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-ingressgateway ISTIO_META_MESH_ID: cluster.local ISTIO_META_POD_NAME: istio-ingressgateway-bd6f57f65-v6zq7 (v1:metadata.name) ISTIO_META_CONFIG_NAMESPACE: istio-system (v1:metadata.namespace) ISTIO_META_ROUTER_MODE: sni-dnat ISTIO_METAJSON_LABELS: {"app":"istio-ingressgateway","istio":"ingressgateway"}

  ISTIO_META_CLUSTER_ID:        Kubernetes
  SDS_ENABLED:                  false
Mounts:
  /etc/certs from istio-certs (ro)
  /etc/istio/ingressgateway-ca-certs from ingressgateway-ca-certs (ro)
  /etc/istio/ingressgateway-certs from ingressgateway-certs (ro)
  /var/run/secrets/kubernetes.io/serviceaccount from istio-ingressgateway-service-account-token-rplrb (ro)

Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: istio-certs: Type: Secret (a volume populated by a Secret) SecretName: istio.istio-ingressgateway-service-account Optional: true ingressgateway-certs: Type: Secret (a volume populated by a Secret) SecretName: istio-ingressgateway-certs Optional: true ingressgateway-ca-certs: Type: Secret (a volume populated by a Secret) SecretName: istio-ingressgateway-ca-certs Optional: true istio-ingressgateway-service-account-token-rplrb: Type: Secret (a volume populated by a Secret) SecretName: istio-ingressgateway-service-account-token-rplrb Optional: false QoS Class: Burstable Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events: `

[adamcarter81]

But just checking - when I direct AGIC to go directly to the application pod (bypassing ISTIO), it works and adds the backend rules with HTTPS - but the readiness probe there is also just http.

[adamcarter81]

@3quanfeng - I just wondered if you would be able to assist me a bit further here? Thanks

[3quanfeng]

Hi @adamcarter81 , you could only probe the port exposed on your container, if you specify the backend protocol https, then you will need to use port 443

[adamcarter81]

Thanks @3quanfeng - just to clarify my situation:

The Istio Ingress gateway is listening on ports 80 & 443 - it will ultimately serve both protocols. AGIC with HTTP to the Istio ingress gateway is fine AGIC with HTTPS to the end pod is fine AGIC with HTTPS to the Isito ingress gateway fails with the backend protocol mismatch.

How does AGIC decide what the backend protocol should be? I am explicitly setting it in the deployment:

`apiVersion: extensions/v1beta1 kind: Ingress metadata: #name: istio-ingressgateway-voting name: agic-ingress-https annotations: kubernetes.io/ingress.class: azure/application-gateway appgw.ingress.kubernetes.io/backend-hostname: "appgw.domain.com" appgw.ingress.kubernetes.io/cookie-based-affinity: "true" appgw.ingress.kubernetes.io/backend-protocol: "Https" appgw.ingress.kubernetes.io/appgw-trusted-root-certificate: wildcard-domain-com-root spec: tls: - hosts: - identityserver-kris.domain.com secretName: wildcard-domain-com

rules:

• host: identityserver-kris.domain.com http: paths:
  • backend: #####serviceName: svc-agic-ingress-https #serviceName: istio-ingressgateway serviceName: istio-ingressgateway servicePort: 443 #path: /* Here it is explicitly set - backend protocol to HTTPS This did work in a previous version of AGIC. The readiness probe of the Isito Ingress Gateway is:kubectl describe pod istio-ingressgateway-d6494d8cd-gqz4p Ports: 15020/TCP, 80/TCP, 443/TCP, 15029/TCP, 15030/TCP, 15031/TCP, 15032/TCP, 15443/TCP, 31400/TCP, 15011/TCP, 15012/TCP, 8060/TCP, 853/TCP, 15090/TCP Readiness: http-get http://:15020/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30 `

Does AGIC use this to build the backend settings, or does it use port 80 if it is found over 443, even though the protocol is specified?

If so, why does exposing the pod to AGIC work, this has the same readiness probe: kubectl describe pod server-deployment-8485574dc9-7p4zp -n test <snip> Port: 443/TCP <snip> Readiness: http-get http://:15020/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30

[3quanfeng]

Hi @adamcarter81 , Thanks for clarification. If so, why does exposing the pod to AGIC work, this has the same readiness probe: kubectl describe pod server-deployment-8485574dc9-7p4zp -n test <snip> Port: 443/TCP <snip> Readiness: http-get http://:15020/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30 Did you specify appgw.ingress.kubernetes.io/backend-protocol: "Https" for the case above?

[adamcarter81]

@3quanfeng yes - see my config : https://github.com/Azure/application-gateway-kubernetes-ingress/issues/875#issuecomment-632873926 appgw.ingress.kubernetes.io/backend-protocol: "Https"

However by adding or removing it, does not change the config to apply to AGIC.

I am still unclear on what is the expected operation: Does AGIC use this to build the backend settings, or does it use port 80 if it is found over 443, even though the protocol is specified, and does AGIC rely on the readiness probe setting to build the back end?

Interestingly the backend was building fine in 1.0, but that had issues with the front end listener

[3quanfeng]

Hi @adamcarter81 , remove or add that annotation won't work for the probe protocol, the scheme you specified in your readiness probe decides the protocol, one benefit is to error out in case of "ApplicationGatewayProbeProtocolMustMatchBackendHttpSettinsProtocol" happens when you try to use https in the httpsettings but probing with http. can you probe istio-gateway at port 443?

[thomas-riccardi]

Some things I finally understood for TLS E2E with istio: AppGW:

• AppGW requires HTTPS probes when backend protocol is HTTPS too (ApplicationGatewayProbeProtocolMustMatchBackendHttpSettinsProtocol error)
• could not find a way to get AppGW debug info on why it returns 502 errors (found nothing useful with diagnostic logs) (on GCP all requests are loggued by default, with a statusDetail: https://cloud.google.com/load-balancing/docs/https/https-logging-monitoring#statusdetail_http_failure_messages)
• if backend hostname is wrongly configured, AppGW probes will still be successful, while all traffic will fail with 502 errors

istio:

• the sidecar automatic injection changes the readiness probes to its own port (otherwise application probes won't work from kubelet), on HTTP, which conflicts with what AppGW expects
  • we can disable that with a pod annotation: sidecar.istio.io/rewriteAppHTTPProbers: "false"
• PeerAuthentication portLevelMtls PERMISSIVE still mangles the traffic (see https://istio.io/latest/docs/concepts/security/#permissive-mode):
  • either it auto detects istio mTLS (possibly with Application Layer Protocol Negociation: next protocol: istio), in which case requires full mTLS (cannot use that with AppGW as client: AppGW cannot setup client authentication)
  • or it assumes plaintext
  • I could not make it accept and not touch normal TLS traffic: curl https: curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number (on the network the istio-proxy returns HTTP/1.1 400 Bad Request to the TLS Client Hello)
• what we can do though is can ask istio not to touch the traffic for some port (without PeerAuthentication) with a pod annotation: traffic.sidecar.istio.io/excludeInboundPorts: "443"
  • then we can have in the pod a plain nginx terminating TLS manually on port 443, without any istio intervention (it's probably possible to use istio sidecar to get certificates, see https://istio.io/latest/blog/2020/proxy-cert/ , didn't try)
  • and set the application readiness probe to that 443 port (httpGet with scheme: HTTPS)
  • finally, set pod annotation appgw.ingress.kubernetes.io/backend-hostname: "myservice" and be sure the manual certificate uses myservice as common name/alternate names), so AppGW stop returning 502 errors

edit: it seems PeerAuthentication portLevelMtls 443 DISABLE does let the traffic pass as-is, functionally equivalent to pod annotation: traffic.sidecar.istio.io/excludeInboundPorts: "443"``. But ideally we would like PERMISSIVE with TLS working by istio-proxy (and not a manual nginx ssl behind), either with full mTLS or if client didn't ask it, just do a ~normal TLS session (still by istio-proxy) instead of returning HTTP/1.1 400. It seems to be discussed last year there: https://discuss.istio.io/t/create-self-signed-classic-tls-cert-with-istio-mtls-enabled/3354/2

edit 2: see https://github.com/istio/istio/issues/14130 and https://github.com/istio/istio/issues/6384#issuecomment-398259867 : it maybe doable to have istio-proxy terminate standard TLS when no ALPN istio sent, using EnvoyFilter to pass the config there?

Result:

• k8s external: AppGW -> nginx https 443 with manual certificates and correct hostnames and no istio mangling at all
• k8s internal: full istio mTLS between all pods
• probably readinessProbes are not perfect: they say OK when istio-proxy may not be ready yet: no graceful update :(

Related issues:

• #906
• #907

[adamcarter81]

@thomas-riccardi Thanks for the information - did you test or were you able to get TLS pass through using AppGW -> Istio Ingress Gateway?

[thomas-riccardi]

@adamcarter81 I did not test AppGW -> Istio Ingress as it seemed to be quite a hack: if I understood correctly to get a native Ingress on top of the Istio Ingress we need to make the native Ingress point to the istio-internal istio-ingressgateway service, then define the Istio Ingress and VirtualService, even the data path seemed too complicated. Furthermore I was afraid it would hit again the readiness probe issue : AGIC would forward the readiness probe of the istio-ingressgateway service pods to the AppGW and possibly have a fatal mismatch on protocol. That being said the pod doesn't seem to have any readiness probe, so not sure how it behaves, maybe it could work (but maybe it would fallback to / as path to probe, which could fail depending on the workload). Anyway, going that way it would be simpler to just remove the AppGW and only use the istio ingress (albeit the network plan would be different..)

[adamcarter81]

Thanks @thomas-riccardi I have it working fine with AppGW --> Isito with HTTP, but it is failing with the HTTP/HTTPS protocol backend check. A custom health probe would solve this. Ultimately I want both HTTP & HTTPS traffic to all ingress via AppGW -> Istio Ingress for full visibility of all traffic in the mesh. The reason for using AppGW was that we hoped to use the Web Application Firewall for added security. I guess we could consider offloading this to an external provider, then bypassing the AppGW

I guess we'll just have to bypass istio ingress for HTTPS now, then revisit it later :/

Thanks for your help & feedback

[ikewabo]

THANK YOU @adamcarter81 and @thomas-riccardi . I'm facing the same dilemma. Many late nights with the same deafening result! Can both ISTIO and MS Azure announce that this is impossible and lets move on? That will save us all the wasted energy.

[abhioncbr]

Hi, I am facing the same issue with the istio-ingress gateway. App gw ingress controller trying for HTTPS readiness probe and failing with Code="ApplicationGatewayProbeProtocolMustMatchBackendHttpSettinsProtocol". SInce istio ingress gateway readiness probe is based on HTTP, and hence getting the error mentioned above. Any plan or update for the issue fixture?

Also, I tried to run a side-car container in the istio-ingress gateway pod to serve the HTTPS; still, it is reporting the same issue. I did remove the readiness probe from the primary container, and now it is giving 404 backend health probe failure on Application Gateway. Do agic supports side car based readiness probe?

To add further, I got it the same setup working with the Nginx ingress controller with both HTTP & HTTPS.

[XBeg9]

Any resolution to this so far? Is it possible to get AGIC -> Istio E2E TLS?

[darthmolen]

Curious myself. Any resolution?

[TeamDman]

Still a problem 2022-07-05. Gateway doesn't respect hostname in the health probe leading the ingress to serve the self signed cert

[karlschriek]

Has any tried an approach using an E2E App Gateway setup that terminates at a reverse proxy, which is itself within the istio service mesh (has a sidecar) and which routes traffic to the actual intended backend server with mTLS enabled?

I mean, I would love to instead see a more native solution like this: AppGW -> Backend Pod (My Server). Traffic is intercepted at backend Pod by istio sidecar and decrypted as though it where mTLS traffic (i.e. give App Gateway the CA cert that that istio uses for mTLS). It feels like it should be possible with some dedicated effort on both projects to get this to work.

But if nothing can come of that, maybe this would also be possible: AppGW -> Backend Pod (Reverse Proxy) -> My Server . In this setup, one would set up for example an NGINX reverse proxy as the backend that the AppGW points to. E2E encryption terminates there. Both the reverse proxy and "My Server" both are then both within the istio service mesh, with mTLS enabled. As long as you can bypass mTLS for the AppGW -> Backend Pod (Reverse Proxy) part this might work.

[karlschriek]

Upon closer inspection I realised that what @thomas-riccardi was talking about above was very similar to what I had in mind, with the exception that I was planning to roll out reverse proxies in their own Pods, so that the hop to the actual backend goes via mTLS, but I don't really think that is necessary, so I will use reverse proxy sidecars instead.

I have now tested out an approach that works as follows:

In words:

• Everything is within the Istio service mesh (i.e. every pod gets an istio-proxy sidecar), with the exception of specific ports that are used as App Gateway backends.
• Any service that we want to expose via an Ingress (with the App Gateway) gets an nginx proxy that does TLS termination. On that Pod, we set the traffic.sidecar.istio.io/excludeInboundPorts: "8443" and sidecar.istio.io/rewriteAppHTTPProbers: "false" annotations
• We enable "frontend" TLS using cert manager
• We enable E2E encryption (i.e. "backend" TLS) with a self-signed CA cert that we install into App Gateway and upload as a Secret into the namespace where we want to mount it as a volume on any backend Pods
• We enable STRICT mTLS

To prove that this works, we roll out two deployments:

• e2e-exposed-server gets an nginx reserve-proxy that listens on port 8443 and does termination of the "backend" TLS, then proxies to "localhost:8080", which is where our actual server lives. This server in turn communicates with...
• e2e-worker-server which serves an httpbin container.

NOTE, in the code below you'll see that in e2e-exposed-server we have a proxy container (i.e. the reverse proxy that does TLS termination and then proxies to localhost:8080) as well as a server container (which is reachable at localhost:8080). Don't be confused by the fact that this server container is also just running nginx. Yes, we are setting up yet another reverse proxy here (this time proxying to http://e2e-worker-server:80/) , but this is not actually a core part of the solution. It is just a convenient way to show that traffic between the e2e-exposed-server and e2e-worker-server Pods will travel via the Istio service mesh (and will therefore be subject to AuthorizationPolicy, PeerAuthentication etc. rules).

Then, we enable PeerAuthentication in STRICT mode and set up AuthorizationPolicies.

In the end we have a solution where:

1. Traffic between browser and "frontend" (e.g. e2e-exposed-server.my-domain.com) is encrypted using the cert issued by cert-manager. (I don't actually show this entirely in the code below since it is well-documented how to do this)
2. Traffic between App GW and "backend" (Pod e2e-exposed-server/proxy:8443) is encrypted with appgw-backend-tls cert
3. Traffic between e2e-exposed-server and e2e-worker-server is encrypted (mTLS) and subject to mesh rules (we are able to block traffic with a "DENY" auth policy for example)

In Pictures:

In Code:

To quickly hack together a prototype that shows the above, here is what you need:

NOTE, there are several placeholders for you to set, such as my-namespace, my-domain.com, my-gateway-name, my-gateway-resource-group etc..

Backend cert

# create backend cert
openssl ecparam -out appgw-backend.key -name prime256v1 -genkey
openssl req -new -sha256 -key appgw-backend.key -out appgw-backend.csr -subj "/CN=appgw-backend"
openssl x509 -req -sha256 -days 365 -in appgw-backend.csr -signkey appgw-backend.key -out appgw-backend.crt

# upload to secret
kubectl create secret tls appgw-backend-tls --key="appgw-backend.key" --cert="appgw-backend.crt" -n my-namespace

# upload to app-gw
az network application-gateway root-cert create \
    --gateway-name "my-gateway-name"  \
    --resource-group "my-gateway-resource-group" \
    --subscription "xxxx-xxxxx-xxx-xxx-xxxxxxxx" \
    --name appgw-backend-tls \
    --cert-file appgw-backend.crt

e2e-exposed-server.yaml

kubectl apply -f e2e-exposed-server.yaml -n my-namespace

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: e2e-exposed-server
spec:
  selector:
    matchLabels:
      app: e2e-exposed-server
  action: DENY
  rules: # deny all traffic. Only traffic on ports set in "traffic.sidecar.istio.io/excludeInboundPorts" annotation will be let through.
    - {}
---
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: e2e
spec:
  mtls:
    mode: STRICT
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: e2e-exposed-server
---
apiVersion: v1
kind: Service
metadata:
  name: e2e-exposed-server
spec:
  selector:
    app: e2e-exposed-server
  ports:
    - protocol: TCP
      port: 8443
      targetPort: 8443
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: e2e-exposed-server
spec:
  selector:
    matchLabels:
      app: e2e-exposed-server
  replicas: 1
  template:
    metadata:
      annotations:
        sidecar.istio.io/rewriteAppHTTPProbers: "false"
        traffic.sidecar.istio.io/excludeInboundPorts: "8443"
      labels:
        app: e2e-exposed-server
    spec:
      serviceAccountName: e2e-exposed-server
      containers:
        # proxy sidecar
        - name: proxy
          imagePullPolicy: Always
          image: nginx:latest
          ports:
            - containerPort: 8443
          volumeMounts:
            - mountPath: /etc/nginx/ssl
              name: secret-volume-proxy
            - mountPath: /etc/nginx/conf.d
              name: configmap-volume-proxy

        # actual server
        - name: server
          imagePullPolicy: Always
          image: nginx:latest
          ports:
            - containerPort: 8080
          volumeMounts:
            - mountPath: /etc/nginx/conf.d
              name: configmap-volume-server
      volumes:
        - name: secret-volume-proxy
          secret:
            secretName: appgw-backend-tls
        - name: configmap-volume-proxy
          configMap:
            name: e2e-exposed-server-proxy
        - name: configmap-volume-server
          configMap:
            name: e2e-exposed-server

---
apiVersion: v1
kind: ConfigMap
metadata:
  name: e2e-exposed-server-proxy
data:
  # This cm is key to having the E2E encryption work. It configures an nginx-sidecar that terminates TLS on port 8443, then
  # proxies to the "server" (which can be any server at all) that listens on the port specified in "proxy_pass"; in this case 8080.
  default.conf: |-
    server {
        listen 8443 ssl;
        root /usr/share/nginx/html;
        index index.html;
        ssl_certificate /etc/nginx/ssl/tls.crt;
        ssl_certificate_key /etc/nginx/ssl/tls.key;
        location / {
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header Host $http_host;
          proxy_pass http://127.0.0.1:8080/;
        }
      }
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: e2e-exposed-server
data:
  # This configmap is not a key part of the pattern, we are just using it to for convenience to configure 
  # the "server" as another nginx proxy (this time one that talks to a server running in a different Pod)
  # as a simple way of proving that inter-pod traffic stays in the service mesh.
  default.conf: |-
    server {
        listen 8080 default_server;
        location / {
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
          proxy_set_header Host $http_host;
          proxy_pass http://e2e-worker-server:80/;
        }
      }
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: e2e-exposed-server
  annotations:
    cert-manager.io/issuer: "my-issuer" #  or use: cert-manager.io/cluster-issuer: "my-cluster-issuer"
    appgw.ingress.kubernetes.io/ssl-redirect: "true"
    appgw.ingress.kubernetes.io/backend-protocol: "https"
    appgw.ingress.kubernetes.io/backend-hostname: "appgw-backend"
    appgw.ingress.kubernetes.io/appgw-trusted-root-certificate: "appgw-backend-tls"
spec:
  ingressClassName: azure-application-gateway # this depends on what you called the ingressClass when you installed AGIC
  tls:
    - hosts:
        - e2e-exposed-server.my-domain.com
      secretName: "e2e-exposed-server-frontend-tls"
  rules:
    - host: e2e-exposed-server.my-domain.com
      http:
        paths:
          - path: "/"
            pathType: Prefix
            backend:
              service:
                name: e2e-exposed-server
                port:
                  number: 8443

e2e-worker-server.yaml

kubectl apply -f e2e-exposed-server.yaml -n my-namespace

apiVersion: apps/v1
kind: Deployment
metadata:
  name: e2e-worker-server
spec:
  selector:
    matchLabels:
      app: e2e-worker-server
  replicas: 1
  template:
    metadata:
      labels:
        app: e2e-worker-server
    spec:
      containers:
        - image: docker.io/kennethreitz/httpbin
          name: httpbin
          ports:
            - containerPort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: e2e-worker-server
spec:
  selector:
    app: e2e-worker-server
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: e2e-worker-server
spec:
  selector:
    matchLabels:
      app: e2e-worker-server
  action: DENY
  rules: # deny all, except traffic coming from Pods that have the "e2e-exposed-server" ServiceAccount attached to them
    - from:
        - source:
            notPrincipals:
              - cluster.local/ns/my-namespace/sa/e2e-exposed-server

Prove traffic between e2e-exposed-server and e2e-worker-server within mesh

Change the e2e-worker-server AuthorizationPolicy to deny all traffic:

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: e2e-worker-server
spec:
  selector:
    matchLabels:
      app: e2e-worker-server
  action: DENY
  rules:
    - {}

If you navigate to e2e-worker-server.my-domain.com you should now see "502 Bad Gateway" or "RBAC access denied"

Prove that mTLS is really working

This is more tricky. You could try following one of these guides:

• https://istio.io/latest/docs/tasks/security/authentication/mtls-migration/
• https://blog.posedio.com/blog/verifying-mtls-in-an-istio-enabled-service-mesh

Below is how I did it:

# for this to work, sidecars have to be injected with "privileged" mode enabled. Set "values.global.proxy.privileged=true" in your istio installation
kubectl exec deployment/e2e-exposed-server -c istio-proxy -n my-namespace -- sudo tcpdump dst port 80 -A > e2e-exposed-server.cap

kubectl exec deployment/e2e-worker-server -c istio-proxy -n my-namespace -- sudo tcpdump dst port 80 -A > e2e-worker-server.cap

# exec into "e2e-exposed-server/server" container
kubectl exec deployment/e2e-exposed-server /bin/bash -c server -it -n my-namespace
> curl http://e2e-worker-server:80

Now close all three sessions. You should now have two files locally, e2e-exposed-server.cap and e2e-worker-server.cap. You can interrogate these yourself manually, or import them in Wireshark to confirm that the traffic is in fact encrypted.