Add option to handle 'dead letters' by sending them to a dead letter queue rather than a dead letter topic [moved from dapr/dotnet-sdk]

[WillPilgrim]

I had some trouble getting the behaviour I wanted from the pubsub component regarding handling 'dead letters'. I'm using RabbitMQ as the pubsub tech. I expected when I forced a drop from the consumer (by sending '"status": "drop"' in the reply) that it would immediately send the message to a RabbitMQ dead letter queue. Instead I found I could configure it to post the message to another topic. In my case I called it deadlettertopic and I sent all message types that were programmatically dropped (or exceeded their retry policy) to that topic. Within the topic consumer I sent these messages (including the cloud events data by NOT using .UseCloudEvents) to a specific queue in RabbitMQ via dapr bindings. It would be much nicer if I could configure either the pubsub component or the consumer to send all dead letters to a nominated queue directly. I realise that different implementations of the pubsub component have different capabilities but some 'do this if supported' capability would be nice.

[yaron2]

Does this help? https://docs.dapr.io/developing-applications/building-blocks/pubsub/pubsub-deadletter/

[berndverst]

Dapr implements deadletter topics client-side (in the sidecar) only (not in the pubsub service through configuration itself) - the Dapr Sidecar lets you configure a Dapr deadletter topic/queue - which from the perspective of the PubSub component is really just a regular Topic or Queue. Under the hood this means the Dapr publishes/enqueues a new message.

This is the feature that @yaron2 describes.

[berndverst]

@WillPilgrim could you reply here? Without further clarification I will be closing this issue in the next two weeks or so.

[WillPilgrim]

Perhaps I am not understanding the mechanism correctly. I supposed that once a message had exceeded the retry policy (or was dropped) and was destined for the dead letter queue, it was sent to a particular endpoint specified in the consuming client. But, if I understand you correctly, it is instead sent to the pubsub component (RabbitMQ in my case) and then delivered to the subscribing endpoint. If that is so then my problem is this: the RabbitMQ queue is only created if I define a subscription in the consuming client. Then, of course, the endpoint is invoked. If I DON'T define a consumer, no queue is created in Rabbit and the dead letters are lost. What I'd like to happen is that I want messages to queue up in RabbitMQ without anyone consuming them - at least not by a subscribing consumer. I want an out of bounds mechanism (potentially manual) to access the dead letters at a later point in time.

[berndverst]

@WillPilgrim The Dapr mechanism itself will after a certain number of failures send the message to a second topic / queue (which doesn't need to have any subscribers bound to it at all. Dapr does not subscribe to this.

So these "failed" messages will in fact queue up in RabbitMQ for later investigation. They won't be delivered anywhere else, unless you manually subscribe to the same queue name you used as your deadletter queue name somewhere else (with or without Dapr).

Please take a look at the link Yaron shared. The deadlettertopic property tells Dapr to send failed messages to this topic... but it will not subscribe to this topic anywhere.

Depending on the pubsub broker of choice you may need to manually create the topic to be used beforehand. I am not sure how your RabbitMQ set up works here.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged (pinned, good first issue, help wanted or triaged/resolved) or other activity occurs. Thank you for your contributions.

[github-actions[bot]]

This issue has been automatically closed because it has not had activity in the last 37 days. If this issue is still valid, please ping a maintainer and ask them to label it as pinned, good first issue, help wanted or triaged/resolved. Thank you for your contributions.