AGIC 1.5.2 can't apply Azure Application gateway when using kubernetes-kubedashboard https with cert-manager

[kittisuw]

Version: Kubernetes v1.24.0 Kubernetes-dashboard v.2.6.1(https://raw.githubusercontent.com/kubernetes/dashboard/v2.6.1/aio/deploy/recommended.yaml) cert-manager cert-manager v1.9.0 mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.2

Currently, I'm using cert-manager v1.9.0 with Kubernetes v. 1.24 with Azure application gateway ingress controller(AGIC). But I got error "azure/application-gateway Unable to find the secret associated to secretId..." when applying kubeernetes-dashboard deployment , ingress rules and look AGIC log seem AGIC can't apply Azure Application gateway. pls help

Note : For some of deployments can apply without any problem eg. https://raw.githubusercontent.com/Azure/application-gateway-kubernetes-ingress/master/docs/examples/aspnetapp.yaml

Ingress rule

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: kdashboard-ingress
  namespace: kubernetes-dashboard
  annotations:
    kubernetes.io/ingress.class: azure/application-gateway
    cert-manager.io/cluster-issuer: "letsencrypt-prod"
    appgw.ingress.kubernetes.io/ssl-redirect: "true"
spec:
  tls:
    - hosts:
      - kdashboard.xxx.com
      secretName: kdashboard-tls
  rules:
  - host: "kdashboard.xxx.com"
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: kubernetes-dashboard
            port:
              number: 443

k get svc

NAME                        TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)    AGE
dashboard-metrics-scraper   ClusterIP   10.0.206.58   <none>        8000/TCP   6h45m
kubernetes-dashboard        ClusterIP   10.0.81.199   <none>        443/TCP    6h45m
---
k describe ing                                            
Name:             kdashboard-ingress
Labels:           <none>
Namespace:        kubernetes-dashboard
Address:          20.195.36.140
Ingress Class:    <none>
Default backend:  <default>
TLS:
  kdashboard-tls terminates kdashboard.xxx.com
Rules:
  Host                      Path  Backends
  ----                      ----  --------
  kdashboard.xxx.com
                            /   kubernetes-dashboard:443 (10.20.0.240:8443)
Annotations:                appgw.ingress.kubernetes.io/ssl-redirect: true
                            cert-manager.io/cluster-issuer: letsencrypt-prod
                            kubernetes.io/ingress.class: azure/application-gateway
Events:
  Type     Reason             Age                From                       Message
  ----     ------             ----               ----                       -------
  Normal   CreateCertificate  26m                cert-manager               Successfully created Certificate "kdashboard-tls"
  Warning  SecretNotFound     11m (x4 over 26m)  azure/application-gateway  Unable to find the secret associated to secretId: [kubernetes-dashboard/kdashboard-tls]

Secret and certificate

k get secrets                                               
NAME                              TYPE     DATA   AGE
kdashboard-tls-bf6pp              Opaque   1      33m

k get cert                                                
NAME             READY   SECRET           AGE
kdashboard-tls   False   kdashboard-tls   32m

AGIC log

k logs -f ingress-appgw-deployment-7dc7f56f8f-mrdwt -n kube-system

-- App Gwy config --        "3"
-- App Gwy config --    ]
-- App Gwy config --}
I0920 09:17:37.921870       1 mutate_app_gateway.go:177] BEGIN AppGateway deployment
I0920 09:17:38.101546       1 mutate_app_gateway.go:183] END AppGateway deployment
E0920 09:17:38.101639       1 controller.go:141] network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="ApplicationGatewayProbeProtocolMustMatchBackendHttpSettinsProtocol" Message="Probe /subscriptions/246ab921-d783-4f40-888d-80dd87f2f723/resourceGroups/bf-hub-rg/providers/Microsoft.Network/applicationGateways/appgw-bf/probes/pb-kubernetes-dashboard-kubernetes-dashboard-443-kdashboard-ingress protocol (Https) does not match Backend Http Setting /subscriptions/246ab921-d783-4f40-888d-80dd87f2f723/resourceGroups/bf-hub-rg/providers/Microsoft.Network/applicationGateways/appgw-bf/backendHttpSettingsCollection/bp-kubernetes-dashboard-kubernetes-dashboard-443-8443-kdashboard-ingress protocol (Http)." Details=[]
E0920 09:17:38.101655       1 worker.go:62] Error processing event.network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="ApplicationGatewayProbeProtocolMustMatchBackendHttpSettinsProtocol" Message="Probe /subscriptions/246ab921-d783-4f40-888d-80dd87f2f723/resourceGroups/bf-hub-rg/providers/Microsoft.Network/applicationGateways/appgw-bf/probes/pb-kubernetes-dashboard-kubernetes-dashboard-443-kdashboard-ingress protocol (Https) does not match Backend Http Setting /subscriptions/246ab921-d783-4f40-888d-80dd87f2f723/resourceGroups/bf-hub-rg/providers/Microsoft.Network/applicationGateways/appgw-bf/backendHttpSettingsCollection/bp-kubernetes-dashboard-kubernetes-dashboard-443-8443-kdashboard-ingress protocol (Http)." Details=[]
I0920 09:17:38.102445       1 event.go:282] Event(v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"ingress-appgw-deployment-7dc7f56f8f-mrdwt", UID:"0f4bcd83-f167-4b76-9a0c-1972e572819e", APIVersion:"v1", ResourceVersion:"5281475", FieldPath:""}): type: 'Warning' reason: 'FailedApplyingAppGwConfig' network.ApplicationGatewaysClient#CreateOrUpdate: Failure sending request: StatusCode=0 -- Original Error: Code="ApplicationGatewayProbeProtocolMustMatchBackendHttpSettinsProtocol" Message="Probe /subscriptions/246ab921-d783-4f40-888d-80dd87f2f723/resourceGroups/bf-hub-rg/providers/Microsoft.Network/applicationGateways/appgw-bf/probes/pb-kubernetes-dashboard-kubernetes-dashboard-443-kdashboard-ingress protocol (Https) does not match Backend Http Setting /subscriptions/246ab921-d783-4f40-888d-80dd87f2f723/resourceGroups/bf-hub-rg/providers/Microsoft.Network/applicationGateways/appgw-bf/backendHttpSettingsCollection/bp-kubernetes-dashboard-kubernetes-dashboard-443-8443-kdashboard-ingress protocol (Http)." Details=[]