The status of Route/Ksvc is not updated even when KCert is updated

[nak3]

In what area(s)?

/area networking

What version of Knative?

HEAD

Steps to Reproduce the Problem

1. Deploy Knative with autoTLS.

istioctl install -y
kubectl apply -f ${SERVING_REPO}/third_party/cert-manager-latest/

kubectl apply --filename https://storage.googleapis.com/knative-nightly/serving/latest/serving-crds.yaml
kubectl apply --filename https://storage.googleapis.com/knative-nightly/serving/latest/serving-core.yaml
kubectl apply --filename https://storage.googleapis.com/knative-nightly/net-istio/latest/release.yaml

kubectl create ns serving-tests
kubectl apply -f ${SERVING_REPO}/test/config/autotls/certmanager/caissuer/
kubectl patch cm config-network -n "knative-serving" -p '{"data":{"autoTLS":"Enabled"}}'

2. Modify config-certmanager with the invalid value

Note: This is a simple step to produce the issue. The same issue happens with some other reasons like let's encrypt issued the certificates slowly.

$ kubectl edit cm -n knative-serving  config-certmanager

BEFORE:

  issuerRef: |
    kind: ClusterIssuer
    name: ca-issuer

AFTER

  issuerRef: |
    kind: ClusterIssuer
    name: INVALID

3. Create Ksvc (it hangs due to invalid cert)

$ kn service create hello-example --image=gcr.io/knative-samples/helloworld-go
  ...
  3.607s Certificate default.example.com is not ready.

The status of Ksvc and route are CertificateNotReady as expected.

$ kubectl get ksvc,rt
NAME                                        URL                                         LATESTCREATED         LATESTREADY           READY     REASON
service.serving.knative.dev/hello-example   https://hello-example.default.example.com   hello-example-00001   hello-example-00001   Unknown   CertificateNotReady

NAME                                      URL                                         READY     REASON
route.serving.knative.dev/hello-example   https://hello-example.default.example.com   Unknown   CertificateNotReady

4. Fix the cert by reverting step-2

$ kubectl edit cm -n knative-serving  config-certmanager

AFTER

  issuerRef: |
    kind: ClusterIssuer
    name: ca-issuer

5. Check the status

Kcert and King become Ready.

$ kubectl get kcert,king
NAME                                                              READY   REASON
certificate.networking.internal.knative.dev/default.example.com   True    

NAME                                                    READY   REASON
ingress.networking.internal.knative.dev/hello-example   True    

But Ksvc and Route are still CertificateNotReady.

$ kubectl get ksvc
NAME            URL                                         LATESTCREATED         LATESTREADY           READY     REASON
hello-example   https://hello-example.default.example.com   hello-example-00001   hello-example-00001   Unknown   CertificateNotReady

Expected Behavior

• When Kcert and Kingress became ready at step-4, Ksvc should become Ready as well.

Actual Behavior

• The status of Ksvc is not updated at step-4.
• When some events trigger the reconcile loop in controller, the status will be updated.

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.

[zhilingc]

+1 same issue :/ On knative-serving 1.3.2.

[richardvflux]

I'm on knative-serving 1.9.2 and I'm getting this as well.

Name:         test.dlvr-test.dlvr-test.fluxfederation.com
Namespace:    test
Labels:       networking.knative.dev/wildcardDomain=dlvr-test.dlvr-test.fluxfederation.com
Annotations:  networking.knative.dev/certificate.class: cert-manager.certificate.networking.knative.dev
API Version:  networking.internal.knative.dev/v1alpha1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2023-04-05T20:42:18Z
  Generation:          1
  Managed Fields:
    API Version:  networking.internal.knative.dev/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .:
          f:networking.knative.dev/certificate.class:
        f:labels:
          .:
          f:networking.knative.dev/wildcardDomain:
        f:ownerReferences:
          .:
          k:{"uid":"72ed00fb-17a0-47b9-8dc3-d8b3640f3685"}:
      f:spec:
        .:
        f:dnsNames:
        f:secretName:
    Manager:      controller
    Operation:    Update
    Time:         2023-04-05T20:42:18Z
    API Version:  networking.internal.knative.dev/v1alpha1
    Fields Type:  FieldsV1
    fieldsV1:
      f:status:
        .:
        f:conditions:
        f:notAfter:
        f:observedGeneration:
    Manager:      controller
    Operation:    Update
    Subresource:  status
    Time:         2023-04-05T20:42:28Z
  Owner References:
    API Version:           v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Namespace
    Name:                  test
    UID:                   72ed00fb-17a0-47b9-8dc3-d8b3640f3685
  Resource Version:        29874350
  UID:                     4b7db946-1f9a-4296-839d-349ca9752309
Spec:
  Dns Names:
    *.test.dlvr-test.dlvr-test.fluxfederation.com
  Secret Name:  test.dlvr-test.dlvr-test.fluxfederation.com
Status:
  Conditions:
    Last Transition Time:  2023-04-05T20:42:28Z
    Status:                True
    Type:                  Ready
  Not After:               2023-07-04T19:42:22Z
  Observed Generation:     1
Events:
  Type    Reason   Age   From                    Message
  ----    ------   ----  ----                    -------
  Normal  Created  16m   namespace-controller    Created Knative Certificate test/test.dlvr-test.dlvr-test.fluxfederation.com
  Normal  Created  16m   certificate-controller  Created Cert-Manager Certificate test/test.dlvr-test.dlvr-test.fluxfederation.com```

```get route -n test
NAME       URL                                                            READY     REASON
my-nginx   https://my-nginx.test.dlvr-test.dlvr-test.fluxfederation.com   Unknown   CertificateNotReady`

This happened to our entire cluster when we upgraded from 1.7.2 -> 1.9.2, we ended up having to manually delete all the kservices and getting Argo to recreate them which causes the resolve to work. ```

[mpluhar]

Any update on this issue?