Unable to create a Sequence with step which has a deadLetterSink defined in the delivery attribute

[joeb-crux]

Describe the bug I am trying to introduce dead letter queues into some of my Sequences in Knative v1.14.3.

When I add a deadLetterSink to a step in the sequence I see the following error in the events on the Sequence object (I'll include the yaml in a thread).

failed to reconcile subscription resource for step: 0 : subscriptions.messaging.knative.dev "dataset-sequence-brand-new-name-kn-sequence-0" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update

I also see errors logged in the kafka-controller in the knative-eventing ns, that I think might be the root cause....

failed to reconcile subscription resource for step: 0 : subscriptions.messaging.knative.dev "dataset-sequence-brand-new-name-kn-sequence-0" is invalid: metadata.resourceVersion: Invalid value: 0x0: must be specified for an update

I have confirmed that all ref's included in the Sequence CRD already exist and are in a Ready state. I have also confirmed that other event routing and delivery are working correctly. In fact within this sequence, the delivery.retry attributes are functioning as expected. I am only seeing this once I add the deadLetterSink attribute.

Expected behavior I expect that when a deadLetterSink is defined on a step within a Sequence, an event will be sent to the referenced object when their is a failed delivery.

To Reproduce Create a Sequence that has 1 step. That step have a delivery attribute defined with a deadLetterSink.

Ensure each Service/Function referenced by the Sequence, including the one set in the deadLetterSink, have been created, applied, and are in a ready state in your cluster.

Apply the Sequence to your cluster.

This is the yaml I used to originally create this issue (if I remove the deadLetterSink block completely this works without issue).

apiVersion: flows.knative.dev/v1
kind: Sequence
metadata:
  name: dataset-sequence-sad
  namespace: cp-oversight-flow
spec:
  channelTemplate: # Under the hood, the Sequence will create a Channel for each step in the sequence
    apiVersion: messaging.knative.dev/v1beta1
    kind: KafkaChannel
    spec:
      numPartitions: 1
      replicationFactor: 1
  steps:
    - ref:
        apiVersion: serving.knative.dev/v1
        kind: Service
        name: fn-dataset-enrich
      delivery:
        deadLetterSink:
          ref:
            apiVersion: serving.knative.dev/v1
            kind: Service
            name: fn-dataset-sequence-dql
        backoffDelay: PT10S
        backoffPolicy: linear
        retry: 2
        timeout: PT2M
  reply:
    ref:
      kind: Service
      apiVersion: serving.knative.dev/v1
      name: fn-dataset-sentinel

Knative release version v1.4.3

Additional context I am in the process of updating to the latest 1.14.x release to see if my issues stem from a version problem. Will update this issue if I am able to reproduce in 1.14.5.

--------- UPDATE --------- I have tried this with 1.14.5 and 1.15.0. The same behavior is present in all versions with 1 main difference. There are no error messages in the kafka-controller with these versions. The main error message from above is still present though.

[Cali0707]

I think https://github.com/knative/eventing/pull/8124 should fix this, so I opened backport PRs for that

[joeb-crux]

@Cali0707 I see the cherry-pick comments from an hour or so ago. Does that mean this PR will make it's way into one of the up coming releases?

I actually found out this afternoon that I can put that delivery block on the channelTemplate to apply to all steps in the sequence. Didn't initially realize that would possible from the docs, but it got me to into a better place. Long term I'd def like to set it at both levels, so I'll give it a try again once that PR makes it's way into a release. Thanks!

[matzew]

@joeb-crux there was now a 1.14.6 https://github.com/knative/eventing/releases/tag/knative-v1.14.6

[joeb-crux]

Thanks! I'll give it a try in the next couple days.

[Cali0707]

Hey @joeb-crux any updates on if this is resolved now?

[pierDipi]

Closing this issue, reopen if the issue is still present