Can't assign port fp-443 to Private IP Listener already assigned to Public IP Listener

[mhsh64]

Describe the bug A clear and concise description of what the bug is.

I have Ingress resources on a AKS cluster which needs using PrivateIp or PublicIp as their front end IP. I have configured one Ingress using public IP, which was fine, however, when using private IP annotation (appgw.ingress.kubernetes.io/use-private-ip: "true") in another Ingress resource, I am getting the bellow error on AGIC:

E0715 15:13:23.665442 1 frontend_listeners.go:46] Can't assign port fp-443 to Private IP Listener fl-d84d98ab259764775ce8700847d97b66; already assigned to Public IP Listener fl-4db314dd864524bfe3bbc9786686cce0; Will not create listener {FrontendPort:443 HostNames:[ myapp-private.mydomain.com ] UsePrivateIP:true} E0715 15:13:23.665476 1 frontend_listeners.go:46] Can't assign port port_80 to Private IP Listener fl-7ad98dd3ce13c2997644bffc6708100d; already assigned to Public IP Listener fl-658b918ffa5ca2a4cab5e0d72e383d46; Will not create listener {FrontendPort:80 HostNames:[ myapp-private.mydomain.com ] UsePrivateIP:true}

And it did not assign any Listeners on Private IP.

Please advise is this possible using Public and Private frontend IP at the same time for same ports on one AppGW?

Ingress1 apiVersion: extensions/v1beta1 kind: Ingress metadata: name: aspnetapp namespace: ns1 annotations: kubernetes.io/ingress.class: azure/application-gateway appgw.ingress.kubernetes.io/appgw-ssl-certificate: rbccert appgw.ingress.kubernetes.io/ssl-redirect: "true" appgw.ingress.kubernetes.io/use-private-ip: "true" spec: rules:

• host: myapp-private.mydomain.com http: paths:
  • path: / backend: serviceName: aspnetapp servicePort: 80

Ingress2

apiVersion: extensions/v1beta1 kind: Ingress metadata: name: kyvos-network-utility namespace: kyvos-network-utility annotations: kubernetes.io/ingress.class: azure/application-gateway appgw.ingress.kubernetes.io/appgw-ssl-certificate: rbccert appgw.ingress.kubernetes.io/ssl-redirect: "true" spec: rules:

• host: myapp.mydomain.com http: paths:
  • backend: serviceName: myapp servicePort: 80 path: /

AGIC version: 1.2.0-rc2 AppGW: WAF V2

To Reproduce Steps to reproduce the behavior:

Ingress Controller details

• Output of kubectl describe pod <ingress controller> . The pod name can be obtained by running helm list.
• Output of `kubectl logs .
• Any Azure support tickets associated with this issue.

[3quanfeng]

Hi @mhsh64 , correct, right now AGIC doesn't support multiple IPs on the same port.

[mhsh64]

@3quanfeng Thanks So at the moment if we want to configure this, we will need two AppGw and two AGIC on the cluster with enabling watch separate namespaces?

If that is true, what if we want to have two public and private Ingress in one identical namespace?

[mhsh64]

@3quanfeng Sorry again, it will be very hard and maybe not beneficial to deploy two appGw ( and two AGIC) , one for internal and one for public ingress for on AKS. However, if this is the only solution for now, is there any document you can please refer me to deploy two AGIC on a same AKS cluster? I wish we were able to have one appGw for using both public and private frontend IPs.

Thanks

[3quanfeng]

Hi @mhsh64 , AGIC support for the feature is definitely on our roadmap, but we don't have ETA for it given appgw itself doesn't support it now. To deploy two AGICs within one AKS cluster but watching on different namespaces, you will need two appgws, and to map one AGIC to one AppGw, you will need to configure in the AGIC helm values.yaml to have the right match. make sure you have

`kubernetes: watchNamespace:

appgw: environment: AZUREPUBLICCLOUD subscriptionId: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx resourceGroup: myResourceGroup name: myApplicationGateway usePrivateIP: false`

[mhsh64]

@3quanfeng Thanks I deployed two appGws with two AGIC, however it will be really hard to manage, since for each new namespace we have to add them in watched list in AGIC cm. Isn’t there anyway to make this more usable?

[3quanfeng]

Hi @mhsh64 , In the next release version(including rc), AGIC will support to delegate two ingress resources in the same namespace to two different controllers.

[mhsh64]

@3quanfeng Thank you for the update I think it might be also useful rather than using watchNamespace in AGIC configmap, using annotations for Ingress resources to let them know which Ingress Controller they can use. So it will be much easier and manageable to have multiple AGICs, since it is not possible yet to have same ports for public and private frontend IPs.

[3quanfeng]

Thanks @mhsh64 , that's exactly the feature we will release soon to make the ingress class configurable.

[mhsh64]

@3quanfeng Thank you Looking forward to it ☺️

[sbkg0002]

This would be great!

[catay]

Are there some timelines available when we can expect this feature ?

Or is there a workaround where I can force the application gateway to listen on another port (8080) for ingress exposed to the private ip ?

Thanks.

[schlickspringer]

@catay the merge request has been already included to support his. however I'm also waiting the release that includes this merge...

[aelmanaa]

@3quanfeng do you know when the merge will be released (cfr. @schlickspringer's message)

[ezYakaEagle442]

@3quanfeng @mscatyao the AppGW doc states :

Can I use the same port for both public-facing and private-facing listeners? No.

The AGIC doc explains how to use a Private IP, but not how to manage the “public+private” use case and to how to configures different ports for public+private IP , could you please add in the docs if this scenario is supported or not ?

[3quanfeng]

Hi @ezYakaEagle442 , there is a note indicating that "App Gateway doesn't support multiple IPs on the same port"

[ezYakaEagle442]

@3quanfeng thanks foryou reply, but this does not answer the question : how to manage the “public+private” use case with AGIC and to how to configures different ports for public+private IP with AGIC. This is not explained in the docs and the ARM template does not manage that neither

[sbkg0002]

Is there any place where we can track the status of this roadmap item?

[Phrow]

Hi, I'm also interested in either the general availability of "multiple ips on same port in app gateway" or anything that makes configuring two app gateways as a workaround easier. Does anyone know when any of this is going to be available? @3quanfeng maybe? Thank you in advance!

[picardsrcd]

I am also very keen on a resolution or instructions on a workaround. I would like to avoid moving to another ingress controller.

[schlickspringer]

@picardsrcd, @Phrow you can check the latest release. I was able to get it working with public/private IP (different ports).

[picardsrcd]

@picardsrcd, @Phrow you can check the latest release. I was able to get it working with public/private IP (different ports).

@schlickspringer Do you have a sample configuration? Are you using the AKS addon or custom deployment? I am using the AGIC AKS addon on a AKS 1.19.7 cluster. Looks like the managed addon is old!

-- App Gwy config --        "managed-by": "ak8s",
-- App Gwy config --        "managed-by-k8s-ingress": "1.2.1/c6389bc7/2020-09-22-23:31T+0000"

Private Example:

apiVersion: v1
kind: Pod
metadata:
  name: aspnetapp-priv
  labels:
    app: aspnetapp-priv
spec:
  containers:
  - image: "mcr.microsoft.com/dotnet/core/samples:aspnetapp"
    name: aspnetapp-image
    ports:
    - containerPort: 80
      protocol: TCP

---

apiVersion: v1
kind: Service
metadata:
  name: aspnetapp-priv
  labels:
    app: aspnetapp-priv
  # annotations:
  #   service.beta.kubernetes.io/azure-load-balancer-internal: "true"
spec:
  selector:
    app: aspnetapp-priv
  # type: LoadBalancer
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 80

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: aspnetapp-priv
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    # appgw.ingress.kubernetes.io/backend-protocol: http
    appgw.ingress.kubernetes.io/use-private-ip: "true"
    appgw.ingress.kubernetes.io/backend-path-prefix: "/"
    appgw.ingress.kubernetes.io/override-frontend-port: "8080"
spec:
  rules:
  - http:
      paths:
      - path: "/private/asp/*"
        backend:
          serviceName: aspnetapp-priv
          servicePort: 8080

Public Example:

apiVersion: v1
kind: Pod
metadata:
  name: aspnetapp-auth
  labels:
    app: aspnetapp-auth
spec:
  containers:
  - image: "mcr.microsoft.com/dotnet/core/samples:aspnetapp"
    name: aspnetapp-image
    ports:
    - containerPort: 80
      protocol: TCP

---

apiVersion: v1
kind: Service
metadata:
  name: aspnetapp-auth
spec:
  selector:
    app: aspnetapp-auth
  ports:
  - protocol: TCP
    port: 80
    targetPort: 80

---

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: aspnetapp-auth
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    appgw.ingress.kubernetes.io/backend-protocol: http
    appgw.ingress.kubernetes.io/backend-path-prefix: "/"
spec:
  rules:
  - http:
      paths:
      - path: /auth/* # The asterix is required to have rewriting work
        backend:
          serviceName: aspnetapp-auth
          servicePort: 80

[Phrow]

@picardsrcd, @Phrow you can check the latest release. I was able to get it working with public/private IP (different ports).

But this whole thread is about using the same port on public and private IP at the same time, no? Different ports should've always worked.

[angelbulas]

Is there no support for public and private IP using the same port at the same time yet?

[mscatyao]

We're actively working on allowing the same port to be used with the public and private IP. We'll update the thread as we continue to make progress on this capability.

[TrotmanRalph]

Any update on this functionality?

[guerillamos]

Bump - Curious if there are any updates on this?

[matt961]

This limitation is quite strange, seems there should be no technical reason for it not to work, but thanks for looking into it @mscatyao

[mscatyao]

This is a limitation on AppGW that we're still working on. I've added this feature request to the roadmap as "In Progress".

[hieumoscow]

Do you ETA on preview @mscatyao?

[JackStromberg]

Hello Team,

Same port on public / private listener is still in the works for Application Gateway. I understand this item has been outstanding for quite some time, however I can confirm work on this has started and we look forward to being able to share future updates with you yet this year.

Appreciate your patience on this one! Jack