AKS
AKS copied to clipboard
Azure
[BUG] Expose the services in an AKS cluster via only one external IP, letting nginx ingress use load balancer IP
Describe the bug
The intention is to create an AKS cluster and expose the services in it via only one external IP by adding an nginx ingress. The issue is very similar to https://github.com/Azure/AKS/issues/1791. This issue was closed and can no longer be commented on.
@palma21 commented that it would be fixed via an upstream issue, but no link to this issue was provided. The next comment was that is was fixed and: "Will be possible on k8s 1.20" Then the issue was closed.
In the meantime we are on 1.23.12. If this is fixed, how can one make the nginx ingress use the external IP of the Azure Loadbalancer instead or provisioning a new external IP?
To Reproduce
- Create cluster:
az group create --name $(resourceGroup) --location $(region)
az aks create \
--resource-group $(resourceGroup) \
--name $(clusterName) \
--node-vm-size $(machineType) \
--node-count 1 \
--vm-set-type VirtualMachineScaleSets \
--enable-cluster-autoscaler \
--min-count $(minSize) \
--max-count $(maxSize) \
--enable-managed-identity \
--generate-ssh-keys
Successfully creates cluster with LB.
- Get external IP $(eval IP := $(shell kubectl --namespace ingress-nginx get services ingress-nginx -o wide | awk 'NR!=1 {print $$4}'))
Returns same IP shown for LP in portal
- Create nginx ingress with static IP
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm repo update ingress-nginx
helm search repo ingress-nginx
helm install ingress-nginx ingress-nginx/ingress-nginx --namespace ingress-nginx --create-namespace \
--set controller.replicaCount=1 \
--set controller.service.loadBalancerIP=$(IP) \
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-resource-group"=$nodeResourceGroup
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz
- Check ingress-nginx service
$ kubectl --namespace ingress-nginx get services
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.0.63.151 <pending> 80:31654/TCP,443:30271/TCP 23m
ingress-nginx-controller-admission ClusterIP 10.0.79.242 <none> 443/TCP 23m
Remains in pending state forever.
Expected behavior The nginx ingress should use the existing external IP
Environment (please complete the following information):
- Kubernetes Client Version: v1.25.2 Kustomize Version: v4.5.7 Server Version: v1.23.12
"azure-cli": "2.40.0", "azure-cli-core": "2.40.0", "azure-cli-telemetry": "1.0.8", "extensions": { "db-up": "0.2.1" }
Hello @rob2universe
First at all, when you create a AKS cluster the way you have created it comes without a namespace called ingress-nginx. Therefore when you are executing
(eval IP :=(shell kubectl --namespace ingress-nginx get services ingress-nginx -o wide | awk 'NR!=1 {print $$4}'))
It is not clear to me which IP address you are getting. My thoughts are that probably you are getting the public IP address from an already existing kubernetes cluster that is setup as current context at that moment. This an lead to a situation where the LB public IP address is always in pending.
If this is the case you can use the following command to know more about the service status, you should see something on the events section:
kubectl describe service ingress-nginx-controller -n ingress-nginx
I have tried to reproduce what you have explained here. I have executed the cluster creation command:
az group create --name test --location westeurope;
az aks create \
--resource-group test \
--name test \
--node-vm-size Standard_B4ms \
--node-count 1 \
--vm-set-type VirtualMachineScaleSets \
--enable-cluster-autoscaler \
--min-count 1 \
--max-count 2 \
--enable-managed-identity \
--generate-ssh-keys
Then I get the credentials:
az aks get-credentials --name test --resource-group test
This command also sets the current-context to this newly created cluster. To confirm execute kubectl config current-context
As I mentioned before, we don't have any ingress-nxing namespace here:
➜ ~ kubectl get namespaces
NAME STATUS AGE
default Active 5m37s
kube-node-lease Active 5m39s
kube-public Active 5m39s
kube-system Active 5m39s
To know the actual public IP address used by the LoadBalancer that gets created with the AKS cluster, you need to check it on the generated resource group by the managed AKS cluster, to do so you can use the following script in bash:
#To generate the resource group name
REGION=westeurope
RESOURCE_GROUP=test
CLUSTER_NAME=test
GENERATED_RESOURCE_GROUP="MC_"$RESOURCE_GROUP"_"$CLUSTER_NAME"_"$REGION
PUBLIC_IP=$(az network public-ip show --name $(az network lb show --name kubernetes --resource-group $GENERATED_RESOURCE_GROUP --query 'frontendIpConfigurations[0].name' -o tsv) --resource-group $GENERATED_RESOURCE_GROUP --query ipAddress -o tsv)
echo $PUBLIC_IP
After this, you can proceed to deploy the ingress-nginx helm chart, where I suggest you to use:
helm install ingress-nginx ingress-nginx/ingress-nginx --namespace ingress-nginx --create-namespace \
--set controller.service.loadBalancerIP=$PUBLIC_IP \
--set controller.service.externalTrafficPolicy=Local
With the option controller.service.externalTrafficPolicy=Local the traffic you get throw the ingress-nginx will have the real user and not the ingress-nginx pod that is doing the proxy.
Finally you will see the public IP address on the LoadBalancer ingress-nginx-controller service.
➜ ~ kubectl get service ingress-nginx-controller -n ingress-nginx
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
ingress-nginx-controller LoadBalancer 10.0.22.229 20.126.231.179 80:30729/TCP,443:30616/TCP 69s
Hope this guide helps you to clarify your problem.
Hello @rob2universe .
Have you got the chance to check the previous comment I posted two weeks ago ?
Best regards.
Hi @carvido1 - many thanks for your reply and effort! I guess you know how it is when a task is urgent and then loses focus. We had some solution and moved on. However, I still would like to solve this better and will get back to this and updated the ticket with the result.
Hello,
I have the same issue, I found out that changing the ports will resolve this. (no idea why...)
so to reproduce, deploy ingress-nginx using helm with
--set controller.service.ports.http=8080
--set controller.service.ports.https=8443
ingress-nginx-controller LoadBalancer 10.0.15.146 74.234.98.230 8088:30210/TCP,8443:32489/TCP 66m
ingress-nginx-controller-admission ClusterIP 10.0.79.52 <none> 443/TCP 66m
ingress-nginx-controller-metrics ClusterIP 10.0.229.247 <none> 10254/TCP 21m
changing it on an running aks cluster fixes it also in few seconds...
![microsoft-github-policy-service[bot] avatar](https://avatars.githubusercontent.com/in/95686?v=4 "Published by a pub.dev verified publisher"){kind=small}