'SecretNotFound' Unable to find the secret associated to secretId

[nagidocs]

trafficstars

Describe the bug 'SecretNotFound' Unable to find the secret associated to secretId: [staging-ns/tistaging-imnos-tls]

To Reproduce Setup AGIC to auto configure https listener on AppGw form Aks cluster deployed website

Ingress Controller details

• Output of kubectl describe pod <ingress controller> . The pod name can be obtained by running helm list.

❯ kc describe po -n ingress agic-ingress-ingress-azure-6bd9994c7d-22l68
Name:         agic-ingress-ingress-azure-6bd9994c7d-22l68
Namespace:    ingress
Priority:     0
Node:         aks-agentpool-23997xxxxxxx/10.8.0.33
Start Time:   Fri, 15 Apr 2022 13:16:05 +0530
Labels:       app=ingress-azure
              pod-template-hash=6bd9994c7d
              release=agic-ingress
Annotations:  checksum/config: df62d8ddee06c6ac6a010674fd03199c9cff7ce7f7f14eacf2166022617efefd
              prometheus.io/port: 8123
              prometheus.io/scrape: true
Status:       Running
IP:           10.8.0.61
IPs:
  IP:           10.8.0.61
Controlled By:  ReplicaSet/agic-ingress-ingress-azure-6bd9994c7d
Containers:
  ingress-azure:
    Container ID:   containerd://9dcd9e703aad0fa1f68dc9babf76747ad40d555f23db6a315b6b99e375a1015f
    Image:          mcr.microsoft.com/azure-application-gateway/kubernetes-ingress:1.5.1
    Image ID:       mcr.microsoft.com/azure-application-gateway/kubernetes-ingress@sha256:cc131292df265926942e23ca5601a3de66e8feabcb81f705d8f7d84b740f81b6
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Fri, 15 Apr 2022 13:16:11 +0530
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:8123/health/alive delay=15s timeout=1s period=20s #success=1 #failure=3
    Readiness:      http-get http://:8123/health/ready delay=5s timeout=1s period=10s #success=1 #failure=3
    Environment Variables from:
      agic-ingress-cm-ingress-azure  ConfigMap  Optional: false
    Environment:
      AZURE_CLOUD_PROVIDER_LOCATION:  /etc/appgw/azure.json
      AGIC_POD_NAME:                  agic-ingress-ingress-azure-6bd9994c7d-22l68 (v1:metadata.name)
      AGIC_POD_NAMESPACE:             ingress (v1:metadata.namespace)
      AZURE_AUTH_LOCATION:            /etc/Azure/Networking-AppGW/auth/armAuth.json
    Mounts:
      /etc/Azure/Networking-AppGW/auth from networking-appgw-k8s-azure-service-principal-mount (ro)
      /etc/appgw/ from azure (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from agic-ingress-sa-ingress-azure-token-jml6g (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             True
  ContainersReady   True
  PodScheduled      True
Volumes:
  azure:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/kubernetes/
    HostPathType:  Directory
  networking-appgw-k8s-azure-service-principal-mount:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  networking-appgw-k8s-azure-service-principal
    Optional:    false
  agic-ingress-sa-ingress-azure-token-jml6g:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  agic-ingress-sa-ingress-azure-token-jml6g
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:          <none>

• Output of `kubectl logs .

kc logs -n ingress agic-ingress-ingress-azure-6bd9994c7d-22l68 | grep SecretNotFound
I0415 07:46:12.598923       1 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"staging-ns", Name:"ingress-agic-testimnos", UID:"0640878e-d0d2-4f35-b750-a231c267c972", APIVersion:"networking.k8s.io/v1", ResourceVersion:"61846290", FieldPath:""}): type: 'Warning' reason: 'SecretNotFound' Unable to find the secret associated to secretId: [staging-ns/tistaging-imnos-tls]
I0415 07:55:57.417209       1 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"staging-ns", Name:"ingress-agic-testimnos", UID:"0640878e-d0d2-4f35-b750-a231c267c972", APIVersion:"networking.k8s.io/v1", ResourceVersion:"62070917", FieldPath:""}): type: 'Warning' reason: 'SecretNotFound' Unable to find the secret associated to secretId: [staging-ns/tistaging-imnos-tls]

I0415 11:37:23.209931       1 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"staging-ns", Name:"ingress-agic-testimnos", UID:"0640878e-d0d2-4f35-b750-a231c267c972", APIVersion:"networking.k8s.io/v1", ResourceVersion:"62123372", FieldPath:""}): type: 'Warning' reason: 'SecretNotFound' Unable to find the secret associated to secretId: [staging-ns/tistaging-imnos-tls]
I0415 11:52:28.797436       1 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"staging-ns", Name:"ingress-agic-testimnos", UID:"0640878e-d0d2-4f35-b750-a231c267c972", APIVersion:"networking.k8s.io/v1", ResourceVersion:"62126936", FieldPath:""}): type: 'Warning' reason: 'SecretNotFound' Unable to find the secret associated to secretId: [staging-ns/tistaging-imnos-tls]
I0415 12:04:27.108387       1 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"staging-ns", Name:"ingress-agic-testimnos", UID:"0640878e-d0d2-4f35-b750-a231c267c972", APIVersion:"networking.k8s.io/v1", ResourceVersion:"62129700", FieldPath:""}): type: 'Warning' reason: 'SecretNotFound' Unable to find the secret associated to secretId: [staging-ns/tistaging-imnos-tl]
I0415 12:08:24.866024       1 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"staging-ns", Name:"ingress-agic-testimnos", UID:"0640878e-d0d2-4f35-b750-a231c267c972", APIVersion:"networking.k8s.io/v1", ResourceVersion:"62130599", FieldPath:""}): type: 'Warning' reason: 'SecretNotFound' Unable to find the secret associated to secretId: [staging-ns/tistaging-imnos-tls]
I0415 12:11:04.761704       1 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"staging-ns", Name:"ingress-agic-testimnos", UID:"0640878e-d0d2-4f35-b750-a231c267c972", APIVersion:"networking.k8s.io/v1", ResourceVersion:"62130599", FieldPath:""}): type: 'Warning' reason: 'SecretNotFound' Unable to find the secret associated to secretId: [staging-ns/tistaging-imnos-tls]
I0415 12:34:49.570196       1 event.go:282] Event(v1.ObjectReference{Kind:"Ingress", Namespace:"staging-ns", Name:"ingress-agic-testimnos", UID:"0640878e-d0d2-4f35-b750-a231c267c972", APIVersion:"networking.k8s.io/v1", ResourceVersion:"62136764", FieldPath:""}): type: 'Warning' reason: 'SecretNotFound' Unable to find the secret associated to secretId: [staging-ns/tistaging-imnos-tls]

• TLS secret already exist in same NS as AGIC Ingress

  kubectl get secrets/tistaging-imnos-tls -n staging-ns
  
  NAME                  TYPE                DATA   AGE
  tistaging-imnos-tls   kubernetes.io/tls   2      20m

Note -: when I'm using the same tls secret with nginx ingress, it works perfectly!

Can you please help me to understand why AGIC not working my tls secret?

[alemag1986]

did you ever find an answer for this? I'm currently having the same issue...

[nagidocs]

Not yet, actually I'm not working on AGIC these days, if I'll find something will post it soon here.

[ganto]

Same issue here. In our case the error is not happening every time but "regularly" when restarting AGIC.

[ganto]

I found the following merged PR #1405 which is already available via 1.6.0-rc1 tag which fixes the issue for us. Within at least 30 AGIC restarts the "SecretNotFound" error didn't occur anymore.

[rarostibo]

We are still experiencing this issue after migration to chart 1.7.2.

Removal of all ingresses, secrets, secretproviderclasses will make it disappear. But subsequent restart of appgw-ingress will cause the issue to re-appear.