pulsar-client-go icon indicating copy to clipboard operation
pulsar-client-go copied to clipboard

[issue #807] dlq topic producer options

Open zzzming opened this issue 3 years ago • 3 comments

Fixes #807

Motivation

To customize producer options for DLQ topics.

Modifications

Add DLQProducerOptions to consumer's DLQ policy.

Verifying this change

  • [ ] Make sure that the change passes the CI checks.

(Please pick either of the following options)

The existing DLQ test is enhanced to cover customized producer options.

Does this pull request potentially affect one of the following parts:

A new producer options field, DLQProuducerOptions, is introduced for DLQ policy to govern the producer options.

ConsumerOptions{
		Topics:              topics,
		DLQ: &DLQPolicy{
			MaxDeliveries:   3,
			DeadLetterTopic: dlqTopic,
			DLQProducerOptions: ProducerOptions{
				BatchingMaxPublishDelay: 100 * time.Millisecond,
			},
		},
  • Dependencies (does it add or upgrade a dependency): no
  • The public API: yes
  • The schema: no
  • The default values of configurations: yes (the existing lz4 compression type is the default.)
  • The wire protocol: no

Documentation

  • Does this pull request introduce a new feature? (yes)
  • If yes, how is the feature documented? (GoDocs)
  • If a feature is not applicable for documentation, explain why?
  • If a feature is not documented yet in this PR, please create a followup issue for adding the documentation

zzzming avatar Jul 17 '22 18:07 zzzming

DLQ and RLQ are created together, and the same data goes first to RLQ and then to DLQ,so the producer configuration of RLQ also needs to be modified? https://github.com/apache/pulsar-client-go/blob/6a8e7f39aac100a285a2c190186e38b73a5c9d34/pulsar/retry_router.go#L128-L132

ne1llee avatar Jul 19 '22 06:07 ne1llee

Added code to use the same ProducerOptions for both DLQ and RLQ.

DLQ and RLQ are created together, and the same data goes first to RLQ and then to DLQ,so the producer configuration of RLQ also needs to be modified?

https://github.com/apache/pulsar-client-go/blob/6a8e7f39aac100a285a2c190186e38b73a5c9d34/pulsar/retry_router.go#L128-L132

zzzming avatar Jul 19 '22 22:07 zzzming

May I ask which version this pull will be megred in?

ne1llee avatar Jul 27 '22 07:07 ne1llee

@wolfstudy @michaeljmarshall Can you help review this PR? @ne1llee is asking when to merge this?

zzzming avatar Aug 23 '22 16:08 zzzming