KNative 1.1.4: channel trigger did not set status.deadLetterSink

[derekwilliamsliquidx]

Describe the bug When Upgrading from Knative 1.0.4 to 1.1.4 we encounter an issue where all of our subscriptions error with the error channel <kafka_channel_name> did not set status.deadLetterSink. This prevents events from being delivered to their appropriate ksvc. This happens with ALL brokers, injected and manually created. We had this exact same issue with a behavior change in 1.0.4 that required the deadlettersink to resolve. Our deadlettersink is a knative service that stores events to be reprocessed later. The solution then was to set the min-replicas to 1 so it always resolved. This time, the upgrade failed and my entire system went offline.

Expected behavior deadlettersink is resolved. Knative continues operating. Subscriptions are created successfully.

To Reproduce Runb knative 1.0.4 Create a Kafka backed broker. Create a deadlettersink that is a knative service with replicas set to 1. Create a ksvc that creates a subscription to kafka Upgrade to knative 1.1.4. Fail.

Knative release version 1.1.4

Additional context Logs of eventing controller

 
May 25, 2022 11:50:29.816
eventing-controller
Event(v1.ObjectReference{Kind:"Subscription", Namespace:"development", Name:"REDACTED", UID:"cb2c6f9c-33ba-47bf-a43f-e80403606861", APIVersion:"messaging.knative.dev/v1", ResourceVersion:"1982207592", FieldPath:""}): type: 'Warning' reason: 'DeadLetterSinkResolveFailed' channel default-kne-trigger didn't set status.deadLetterSinkURI
  
May 25, 2022 11:50:29.816
eventing-controller
Getting channel
 
May 25, 2022 11:50:29.816
eventing-controller
Channel didn't set status.deadLetterSinkURI
 
May 25, 2022 11:50:29.816
eventing-controller
Returned an event
 
May 25, 2022 11:50:29.816
eventing-controller
Reconcile succeeded
 
May 25, 2022 11:50:29.816
eventing-controller
Event(v1.ObjectReference{Kind:"Subscription", Namespace:"development", Name:"REDACTED", UID:"250a8ce7-0f87-483a-a84c-84ee2ad1955f", APIVersion:"messaging.knative.dev/v1", ResourceVersion:"1982207449", FieldPath:""}): type: 'Warning' reason: 'DeadLetterSinkResolveFailed' channel default-kne-trigger didn't set status.deadLetterSinkURI
 
May 25, 2022 11:50:29.817
eventing-controller
Getting channel

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.