Public IP HTTPS fails when creating private IP HTTPS ingress

[sprayzcs]

Describe the bug When creating a private ingress resource (annotation appgw.ingress.kubernetes.io/use-private-ip: "true"), all public https listeners fail with the error ERR_CONNECTION_RESET (from edge)

To Reproduce

• Create AGW with public and private ip
• Create public ingress with https
• Create private ingress with https

Result: The public https page returns the stated error while the private https page functions normally.
When turning off https on the public page, the public page loads normally.
When turning off https on the pivate page and turning on https on the public page, the public page works again with https.

I am using a custom CA to generate my TLS certificates, if that helps..

Ingress Controller details

• Output of kubectl describe pod <ingress controller> . The pod name can be obtained by running helm list.
• Output of kubectl logs <ingress controller>:

Name:             ingress-appgw-deployment-6b67ffdf9d-rq95z
Namespace:        kube-system
Priority:         0
Service Account:  ingress-appgw-sa
Node:             aks-system0000-37908770-vmss000007/10.1.0.5
Start Time:       Mon, 13 Nov 2023 14:10:14 +0100
Labels:           app=ingress-appgw
                  kubernetes.azure.com/managedby=aks
                  pod-template-hash=6b67ffdf9d
Annotations:      checksum/config: 163d031e33ff447cec536dd3a56a52e88c0a227e1fe6fe40c0112a69b036c212
                  cluster-autoscaler.kubernetes.io/safe-to-evict: true
                  kubectl.kubernetes.io/restartedAt: 2023-11-13T14:10:13+01:00
                  kubernetes.azure.com/metrics-scrape: true
                  prometheus.io/path: /metrics
                  prometheus.io/port: 8123
                  prometheus.io/scrape: true
                  resource-id:
                    /subscriptions/<xxx>/resourceGroups/<xxx>/providers/Microsoft.ContainerService/managedClusters/aks
Status:           Running
IP:               10.0.0.12
IPs:
  IP:           10.0.0.12
Controlled By:  ReplicaSet/ingress-appgw-deployment-6b67ffdf9d
Containers:
  ingress-appgw-container:
    Container ID:   containerd://781c04692103eb02469401588073e0a2c92ba5f9a25ecd34ce8773945052708e
    Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.7.2
    Image ID:       mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:eeb1d42ebfb872478d9b0b16f6936ea938d6e5eed4a59cde332b8757556a5e1f
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Mon, 13 Nov 2023 14:10:14 +0100
    Ready:          True
    Restart Count:  0
    Limits:
      cpu:     700m
      memory:  600Mi
    Requests:
      cpu:      100m
      memory:   20Mi
    Liveness:   http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:  http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      ingress-appgw-cm  ConfigMap  Optional: false
    Environment:
      AGIC_POD_NAMESPACE:             kube-system (v1:metadata.namespace)
      KUBERNETES_PORT_443_TCP_ADDR:   <xxx>.azmk8s.io
      KUBERNETES_PORT:                tcp://<xxx>.azmk8s.io:443
      KUBERNETES_PORT_443_TCP:        tcp://<xxx>.azmk8s.io:443
      KUBERNETES_SERVICE_HOST:        <xxx>.azmk8s.io
      AZURE_CLOUD_PROVIDER_LOCATION:  /etc/kubernetes/azure.json
      AGIC_POD_NAME:                  ingress-appgw-deployment-6b67ffdf9d-rq95z (v1:metadata.name)
    Mounts:
      /etc/kubernetes/azure.json from cloud-provider-config (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-jcb2h (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  cloud-provider-config:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/azure.json
    HostPathType:  File
  kube-api-access-jcb2h:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 CriticalAddonsOnly op=Exists
                             node.kubernetes.io/memory-pressure:NoSchedule op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

• Any Azure support tickets associated with this issue.

[akshaysngupta]

can you check what frontend all public ingresses are using on AppGateway ?

[sprayzcs]

can you check what frontend all public ingresses are using on AppGateway ?

There are two public and two private ingress resources. the first and third are my public ingresses, and the second and fourth are my private ingresses.

The public ingresses are using the public frontend ip.

Edit: I also noticed that shortly after the creation of the private ingress, both (public and private) ingresses function with https. After a short amount of time (~ 30 seconds), the public ingress does not work with https anymore.

[akshaysngupta]

@sprayzcs From the screenshot, is it right to conclude that AppGW config is generated as expected by AGIC but you see connectivity issue with AppGW ? If so, can you please create a support ticket ?