pulsar-client-node icon indicating copy to clipboard operation
pulsar-client-node copied to clipboard

[Feature request] Configure the auto-created DLQ producer

Open dreamerblue opened this issue 1 year ago • 3 comments

We really like pulsar and would like to see more features in the node client.

We found the deadLetterPolicy only provides a few options, and the auto-created producer is non-configurable. We are requesting to add an extra producer config field under ConsumerConfig.deadLetterPolicy that will allow auto-created producers to configure properties such as the maximum number of batching messages, maximum delay to publish and whether to enable batching, etc.

dreamerblue avatar Feb 16 '24 06:02 dreamerblue

Another reason we need this feature:

If we send multiple dead letter messages in a batch then they will have the same ORIGIN_MESSAGE_ID (see the demo below). this is a big difficulty to manually troubleshoot the original message from DLQ. So we need to turn off batching.

image

dreamerblue avatar Feb 16 '24 06:02 dreamerblue

The producer of DLQ should be a low-frequency operation, and we can turn off batch sending by default, just like the Java client behaves. And there's no need to expose those configurations, what do you think?

https://github.com/apache/pulsar/blob/252509e08018eeed189e0d5492c3664a67bced06/pulsar-client/src/main/java/org/apache/pulsar/client/impl/ConsumerImpl.java#L2166-L2189

shibd avatar Feb 18 '24 12:02 shibd

Thanks for your reply. I think it's reasonable. The new client behavior can avoid some potential troubles.

dreamerblue avatar Feb 20 '24 03:02 dreamerblue

Close this issue, it solves by cpp3.5.0, and the node.js client has already upgraded it.

shibd avatar Apr 03 '24 07:04 shibd