App Gateway sslcert not deleted during terraform destroy (also max 100 sslcerts)

[jimusbobus]

Describe the bug We use Helm and Terraform to deploy services into AKS, it seems when we 'terraform destroy' a deployed service the sslcert data that was applied to the App Gateway is not deleted.

To Reproduce Deployed service with an ingress controller, here is the ingress.yaml that is used as part of the deployment

kubectl get ingress -n auto-terratestjdr1 -o yaml

`apiVersion: v1 items:

• apiVersion: extensions/v1beta1 kind: Ingress metadata: annotations: appgw.ingress.kubernetes.io/backend-protocol: https appgw.ingress.kubernetes.io/request-timeout: "60" appgw.ingress.kubernetes.io/ssl-redirect: "true" kubernetes.io/ingress.class: azure/application-gateway creationTimestamp: "2020-08-28T14:34:10Z" generation: 1 labels: netsec.barracuda.com/environment: dev netsec.barracuda.com/name: terratestjdr1 name: terratestjdr1 namespace: auto-terratestjdr1 resourceVersion: "27567478" selfLink: /apis/extensions/v1beta1/namespaces/auto-terratestjdr1/ingresses/terratestjdr1 uid: fdd32ed1-a5b6-4fbe-95b5-016749007494 spec: rules:
  • host: springboot-demo-appjdr1.dev.bnngm.azure.cudaops.com http: paths:
    • backend: serviceName: terratestjdr1 servicePort: 8443 tls:
  • hosts:
    • springboot-demo-appjdr1.dev.bnngm.azure.cudaops.com secretName: star-dev-bnngm-azure-cudaops-com status: loadBalancer: ingress:
    • ip: 51.105.147.67 kind: List metadata: resourceVersion: "" selfLink: ""`

Verify that the cert has been installed

az network application-gateway ssl-cert list -g bnngm-dev-westeurope --gateway-name bnngm-dev-westeurope-aks-appgw --subscription blah-blah | jq

{ "data": null, "etag": "W/\"cf3dcf6a-0fe4-4038-a59a-428923b3362e\"", "id": "/subscriptions/4b7cd783-c55a-4319-a0d7-a3a68ef112b1/resourceGroups/bnngm-dev-westeurope/providers/Microsoft.Network/applicationGateways/bnngm-dev-westeurope-aks-appgw/sslCertificates/auto-terratestjdr1-star-dev-bnngm-azure-cudaops-com", "keyVaultSecretId": null, "name": "auto-terratestjdr1-star-dev-bnngm-azure-cudaops-com", "password": null, "provisioningState": "Succeeded", "publicCertData": "FAKEDATA", "resourceGroup": "bnngm-dev-westeurope", "type": "Microsoft.Network/applicationGateways/sslCertificates" }

Destroy the deployment and check for the cert again; az network application-gateway ssl-cert list -g bnngm-dev-westeurope --gateway-name bnngm-dev-westeurope-aks-appgw --subscription blah-blah | jq

[ { "data": null, "etag": "W/\"738c1667-1f66-4560-ba3e-b28ae25b4460\"", "id": "/subscriptions/4b7cd783-c55a-4319-a0d7-a3a68ef112b1/resourceGroups/bnngm-dev-westeurope/providers/Microsoft.Network/applicationGateways/bnngm-dev-westeurope-aks-appgw/sslCertificates/auto-terratestjdr1-star-dev-bnngm-azure-cudaops-com", "keyVaultSecretId": null, "name": "auto-terratestjdr1-star-dev-bnngm-azure-cudaops-com", "password": null, "provisioningState": "Succeeded", "publicCertData": "FAKEIT", "resourceGroup": "bnngm-dev-westeurope", "type": "Microsoft.Network/applicationGateways/sslCertificates" } ]

The sslcert is not removed with the rest of the deployment.
Once we hit 100 deployments then we cannot deploy anymore.

Ingress Controller details kubectl describe pod -n ingress-azure ingress-azure-69c7c9d66d-sjgkl

Name: ingress-azure-69c7c9d66d-sjgkl Namespace: ingress-azure Priority: 0 Node: aks-agentpool-18118092-vmss000000/10.153.40.4 Start Time: Mon, 27 Jul 2020 21:21:10 +0100 Labels: aadpodidbinding=ingress-azure app=ingress-azure pod-template-hash=69c7c9d66d release=ingress-azure Annotations: checksum/config: 92368a0b96949a0f3e010d229a45caa26fea001a744095ee3f780595cc508397 kubectl.kubernetes.io/restartedAt: 2020-07-15T11:10:45+01:00 prometheus.io/port: 8123 prometheus.io/scrape: true Status: Running IP: 10.153.40.9 IPs: IP: 10.153.40.9 Controlled By: ReplicaSet/ingress-azure-69c7c9d66d Containers: ingress-azure: Container ID: docker://cffae75aa13e593b9d1422ac3c1aa8ea991768bd607af7b8e29fdf8467178e91 Image: mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.2.0 Image ID: docker-pullable://mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:de458f962eab0cd2de19d23dfeb9a0e4bc2565a38f8c45cc98a74f3cda8b940c Port: Host Port: State: Running Started: Mon, 27 Jul 2020 21:21:23 +0100 Ready: True Restart Count: 0 Liveness: http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3 Readiness: http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3 Environment Variables from: ingress-azure ConfigMap Optional: false Environment: AZURE_CLOUD_PROVIDER_LOCATION: /etc/appgw/azure.json AGIC_POD_NAME: ingress-azure-69c7c9d66d-sjgkl (v1:metadata.name) AGIC_POD_NAMESPACE: ingress-azure (v1:metadata.namespace) Mounts: /etc/appgw/azure.json from azure (rw) /var/run/secrets/kubernetes.io/serviceaccount from ingress-azure-token-dhdlh (ro) Conditions: Type Status Initialized True Ready True ContainersReady True PodScheduled True Volumes: azure: Type: HostPath (bare host directory volume) Path: /etc/kubernetes/azure.json HostPathType: File ingress-azure-token-dhdlh: Type: Secret (a volume populated by a Secret) SecretName: ingress-azure-token-dhdlh Optional: false QoS Class: BestEffort Node-Selectors: Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s node.kubernetes.io/unreachable:NoExecute for 300s Events:

terraform -version Terraform v0.12.29

[nboeckmann]

Hi,

I'm experiencing the same issue. I don't think that it classifies as an improvement but must be treated as a bug as the ingress controller simply stops working in my environment after I've added/deleted 100 ingress definitions. I think it's important that the ingress controller cleans up after itself and when it deletes a listener it also has to delete the associated SSL certificate.

Nils

[chreichert]

Any news on this issue. I would also agree, that this classifies as bug. Are there any plans, when this will be fixed?

[gregory-j-baker]

@akshaysngupta -- are you able to comment on this?