Unable to upgrade second AGIC with helm on same cluster

[ferronsw]

Describe the bug I want to upgrade AGIC from 1.4.0 to 1.5.1. I did this with the fist of two AGIC deployments on the cluster. Now I want to upgrade the second one, but I get this error:

Error: UPGRADE FAILED: rendered manifests contain a resource that already exists. Unable to continue with update: IngressClass "azure-application-gateway" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "ingress-azure-development": current value is "ingress-azure"; annotation validation error: key "meta.helm.sh/release-namespace" must equal "development": current value is "default"

To Reproduce Use this helm config for the first AGIC:

# This file contains the essential configs for the ingress controller helm chart

# Verbosity level of the App Gateway Ingress Controller
verbosityLevel: 3

################################################################################
# Specify which application gateway the ingress controller will manage
#
appgw:
    subscriptionId: <subscriptionId>
    resourceGroup: RSG-VRMWB-AKS
    name: VRMWB-AppGateway
    usePrivateIP: false
    waf_listener: true

    # Setting appgw.shared to "true" will create an AzureIngressProhibitedTarget CRD.
    # This prohibits AGIC from applying config for any host/path.
    # Use "kubectl get AzureIngressProhibitedTargets" to view and change this.
    shared: false

################################################################################
# Specify which kubernetes namespace the ingress controller will watch
# Default value is "default"
# Leaving this variable out or setting it to blank or empty string would
# result in Ingress Controller observing all acessible namespaces.
#
kubernetes:
  watchNamespace: productie

################################################################################
# Specify the authentication with Azure Resource Manager
#
# Two authentication methods are available:
# - Option 1: AAD-Pod-Identity (https://github.com/Azure/aad-pod-identity)
##armAuth:
#    type: aadPodIdentity
#    identityResourceID: <identityResourceId>
#    identityClientID:  <identityClientId>

## Alternatively you can use Service Principal credentials
armAuth:
    type: servicePrincipal
    secretJSON: <secret>

################################################################################
# Specify if the cluster is RBAC enabled or not
rbac:
    enabled: true # true/false

# Specify aks cluster related information. THIS IS BEING DEPRECATED.
aksClusterConfiguration:
    apiServerAddress: <dnsname>.privatelink.westeurope.azmk8s.io

Update with:

helm upgrade ingress-azure -f .\agic-helm-config.yaml application-gateway-kubernetes-ingress/ingress-azure --version 1.5.1  --namespace default --set nodeSelector."environment"=productie

Use this helm config for the second AGIC:

# This file contains the essential configs for the ingress controller helm chart

# Verbosity level of the App Gateway Ingress Controller
verbosityLevel: 3

################################################################################
# Specify which application gateway the ingress controller will manage
#
appgw:
    subscriptionId: <subscriptionId>
    resourceGroup: RSG-VRMWB-AKS
    name: VRMWB-AppGateway-Development
    usePrivateIP: false
    waf_listener: false

    # Setting appgw.shared to "true" will create an AzureIngressProhibitedTarget CRD.
    # This prohibits AGIC from applying config for any host/path.
    # Use "kubectl get AzureIngressProhibitedTargets" to view and change this.
    shared: false

################################################################################
# Specify which kubernetes namespace the ingress controller will watch
# Default value is "default"
# Leaving this variable out or setting it to blank or empty string would
# result in Ingress Controller observing all acessible namespaces.
#
kubernetes:
  watchNamespace: development

################################################################################
# Specify the authentication with Azure Resource Manager
#
# Two authentication methods are available:
# - Option 1: AAD-Pod-Identity (https://github.com/Azure/aad-pod-identity)
##armAuth:
#    type: aadPodIdentity
#    identityResourceID: <identityResourceId>
#    identityClientID:  <identityClientId>

## Alternatively you can use Service Principal credentials
armAuth:
    type: servicePrincipal
    secretJSON: <secretJSON>

################################################################################
# Specify if the cluster is RBAC enabled or not
rbac:
    enabled: true # true/false

# Specify aks cluster related information. THIS IS BEING DEPRECATED.
aksClusterConfiguration:
    apiServerAddress: <dnsname>.privatelink.westeurope.azmk8s.io

Trying to update with:

helm upgrade ingress-azure-development -f .\agic-helm-config.yaml application-gateway-kubernetes-ingress/ingress-azure --version 1.5.1  --namespace development --set nodeSelector."environment"=development

Results in:

Error: UPGRADE FAILED: rendered manifests contain a resource that already exists. Unable to continue with update: IngressClass "azure-application-gateway" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "ingress-azure-development": current value is "ingress-azure"; annotation validation error: key "meta.helm.sh/release-namespace" must equal "development": current value is "default"

As you can see in the helm configs I use two different watchNamespaces and two different gateways. I looked at the config map an saw this new values with the already upgraded instance:

	"INGRESS_CLASS_RESOURCE_CONTROLLER": "azure/application-gateway",
	"INGRESS_CLASS_RESOURCE_ENABLED": "true",
	"INGRESS_CLASS_RESOURCE_NAME": "azure-application-gateway",
	"KUBERNETES_WATCHNAMESPACE": "productie",
	"MULTI_CLUSTER_MODE": "false"

I think the INGRESS_CLASS_RESOURCE values are causing the problem. I can add these values manually to the development instance config map, but I don't want to destroy my environment.

[akshaysngupta]

@ferronsw thanks for sharing. Seems like IngressClass resource is having issues with installation. Will work on a fix.

[mkemmerz]

Any updates on this? We also struggle with this issue as we have 2 AGIC for a single cluster.

[mkemmerz]

so we figured out this not so nice workaround to fix this temporarely and to be able to upgrade to 1.5.1 (and then to Kubernetes 1.22):

1. Delete the current IngressClass "azure-application-gateway"

2. Add the watchNamespaces (if not already used) to every AGIC (this is important to make it work later)

3. Every AGIC receives (and creates) its own IngressClass. instance_count is just a counter in this case to guarantee uniqueness (1, 2, 3 ...). ONLY ONE IngressClass is set as true for the value of default.

  ingressClassResource:
    name: azure-application-gateway-${instance_count}
    enabled: true
    default: ${default_ingress_class} // set to true for the first IngressClass, otherwise always false
    controllerValue: "azure/application-gateway"

4. We make use of the default behavior of the IngressClass:

Default IngressClass You can mark a particular IngressClass as default for your cluster. Setting the ingressclass.kubernetes.io/is-default-class annotation to true on an IngressClass resource will ensure that new Ingresses without an ingressClassName field specified will be assigned this default IngressClass.

As all of our Ingress still use kubernetes.io/ingress.class: azure/application-gateway they will automatically use the IngressClass with default set to true.

It doesnt matter which IngressClass is used because they all aim to azure/application-gateway (the same controller). The AGIC is able to seperate the load by using the watchNamespace attribute.

[djam7]

My team is preparing for an AKS upgrade and are currently testing our application functionality on a newly installed (not upgraded) kubernetes cluster at version 1.22.6 with terraform/helm deployed agics using helm chart version 1.5.1. No issues were discovered until we deployed our second namespace which errors out...

Error: rendered manifests contain a resource that already exists. Unable to continue with update: IngressClass "azure-application-gateway" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "agic-pocclient03": current value is "agic-pocclient04"

We deploy one application-gateway per namespace and are explicitly setting watchNamespace.

We hit this error with both 1.5.0 and 1.5.1 releases, but can successfully deploy with 1.5.0-rc1. However for our pending upgrade we are hesitant about moving forward with a release candidate.

How long might it be until a 1.5.2 release with this fix is available?

[johannordincab]

Had the same trouble today with this. Currently running on 1.5.0.rc1 but will wait for a fix before going to out production-cluster with it.

[ferronsw]

@akshaysngupta Can you share an update on the timeline please?

[akshaysngupta]

This is happening as both AGICs are using the same ingress class resource name which will lead to conflict. You can provide a different value for ingress class name by using providing kubernetes.ingressClassResource.name parameter in the Helm values. Remember to use the same name in the Ingress resource as well.

[chienvnguyen]

This is happening as both AGICs are using the same ingress class resource name which will lead to conflict. You can provide a different value for ingress class name by using providing kubernetes.ingressClassResource.name parameter in the Helm values. Remember to use the same name in the Ingress resource as well.

Can you please explain more about Remember to use the same name in the Ingress resource as well.? I'm still facing this issue after providing kubernetes.ingressClassResource.name parameter.

[chienvnguyen]

1. Delete the current IngressClass "azure-application-gateway"

@mkemmerz Can you please share details on how did you do this? :)

[mkemmerz]

1. Delete the current IngressClass "azure-application-gateway"

@mkemmerz Can you please share details on how did you do this? :)

This is a very good question 😥 Unfortunately I don't really remember as I have written this already more than half a year ago. In the worst case just completely replace your currently running AGIC with a new one. I don't think there is any resource created but I am not 100% sure. Maybe just a miss wording from my side. If you find out more and notice that I am wrong please correct me!