AzureIngressProhibitedTarget not working as expected - AG resources still removed by AKS

[marc-mueller]

Describe the bug In our deployment, we have a single service which is run by VMSS and needs to be available through the same application gateway as AKS is using. We added a AzureIngressProhibitedTarget CRD with the domain name of the VMSS service. But after every AKS deployment, the VMSS objects are completely removed by AKS.

To Reproduce Steps to reproduce the behavior:

• Crate a AzureIngressProhibitedTarget in default namespace with domain hostname
• Create the following AG objects by az cli: ssl-cert, address-pool, frontend-port, http-listener, probe, https-settings, rule --> the http-listener is pointing to the hostname
• Do an AKS deployment, the above manually added AG objects are removed

Did I miss something? Do I need to pay special attention to some details while manually creating the AG objects? The only thing mentioned in the documentation is the hostname which has to match.

Thanks for your help!

[3quanfeng]

Hi @marc-mueller , May I ask which version of AGIC you deployed? there is a related bug fixed in v1.2.0-rc3

[marc-mueller]

Hi @3quanfeng, we are currently using 1.0.0.

[3quanfeng]

Hi @marc-mueller , did you have prohibit-all-targets deployed as well before your own prohibited policy? also, could you please kindly share your sanitized prohibited policy?

[marc-mueller]

No, I just deployed my custom prohibited policy. AKS was setup before and then later on we needed the manually managed endpoints in AG. If I understood the documentation correctly, the custom policy is the only thing needed in this case, right?

We used the following policy:

apiVersion: "appgw.ingress.k8s.io/v1"
kind: AzureIngressProhibitedTarget
metadata:
  name: manually-configured-xy-service
spec:
  hostname: xy.mydomain.com

[3quanfeng]

Thanks much @marc-mueller . Could you please help check your prohibited policies by "kubectl get AzureIngressProhibitedTarget". when you upgrade AGIC with shared enabled(assuming you upgrade AGIC with shared enabled via helm), a default prohibit-all-targets policy will be installed, you can then install your own policy as you mentioned above, but to make your policy work, you will need to delete the default one which is prohibit-all-targets. Please let us know if you still run into the same issue by following the steps above(or the doc here). Thanks!

[akshaysngupta]

@marc-mueller Can you please check if the prohibited target is being picked by AGIC by checking the logs ? Also, can you upgrade the installation to 1.2.0 ?

[Vishal2696]

@marc-mueller Which namespace is your AGIC deployed? Is AGIC and prohibited targets in the same namespace? Was AGIC configured to look at all namespaces during installation? Need to clear these out first before going deeper.

[walvarado-applaudo]

Hey guys, i have the exact same problem.

According to the docs, if there are a previously deployed AGIC, using a custom prohibited policy, should be enough.

But as soon as I made a k8s ingress (AGIC) deployment the manually created AG objects are deleted. Do you have any ideas? Could you solve the issue?

[adamcarter81]

Hi all - I have just run into this as well - I have AGIC deployed (1.5.0), but now needed to add a custom hostname not handled by AKS. Easy I thought, I remember AGIC could ignore certain hostnames, and the documentation states for an existing AGIC installation we just need to create a CRD for the hostname. I did in the default namespace (as per AGIC), but sure enough - on the next refresh it just deletes everything associated with it (listener, rules, backend etc).

Am I missing a step somewhere?

[adamcarter81]

I think I have solved this - I still needed to set the shared=true in the helm values - once I did this and deleted the prohibitall CRD it kept my custom domain. The documentation should be updated for the section Enable for an existing AGIC installation - that still the shared setting needs to be updated

[manikanta1994]

Hi Team,

I had velero in my aks cluster, When the AKS cluster is at 1.21.9 version everything worked fine, We had upgraded to 1.22.4 -->1.23.5. Present cluster version is 1.23.5. Once after upgrade my backups are failing due to below error, Could some one please look into this and help me on this ?

[juanpgarces]

I followed the documentation and the same result happened to me. In the next refresh it just deletes everything that was added manually that referenced the hostname which was supposed to be a Prohibited Target. We installed AGIC through the AKS-Addon so I am unsure of how to enable the "shared" valued @adamcarter81 was referring to. Does anyone have more information/documentation on a way to enable this for an AGIC installed through the AKS-Addon? Any help is appreciated.

Also, if that does fix the issue the documentation needs to be updated to state that setting has to be enabled.

[shansted]

I went down this path today, and tracked this down to a configmap setting the helm chart sets.

https://github.com/Azure/application-gateway-kubernetes-ingress/blob/master/helm/ingress-azure/templates/configmap.yaml

APPGW_ENABLE_SHARED_APPGW: 'true'

After setting this, check for the following entries in the AGIC pod logs.

[brownfield] Existing Blacklisted Rules AGIC will retain: [brownfield] Existing Rules AGIC will remove:

These were not present before setting the flag.

[bjburke002]

I went down this path today, and tracked this down to a configmap setting the helm chart sets.

https://github.com/Azure/application-gateway-kubernetes-ingress/blob/master/helm/ingress-azure/templates/configmap.yaml

APPGW_ENABLE_SHARED_APPGW: 'true'

After setting this, check for the following entries in the AGIC pod logs.

[brownfield] Existing Blacklisted Rules AGIC will retain: [brownfield] Existing Rules AGIC will remove:

These were not present before setting the flag.

Thanks for this--couldn't find it for the life of me but this cleared up an issue I was dealing with.