pulsar icon indicating copy to clipboard operation
pulsar copied to clipboard

[improve] [test] add end to end deduplication test.

Open thetumbled opened this issue 1 year ago • 1 comments

Motivation

Message deduplication feature has been introduced into Pulsar long time ago, but the concrete problem it solve is vague. The info i can get from the official document is that we can enable a switch in the conf file, and then we can enjoy the message deduplication feature. But actually, there are a lot of point that we don't clarify.

  1. How do we use it? Should user set the sequence id by api exposed manually, or just let the pulsar client mange it only?
  2. Which concrete situation will we meet message duplication in? Can message deduplication feature handle these situations all?

Modifications

Add end to end deduplication test to clarify what problem message deduplication can handle, and what it can't.

Verifying this change

  • [x] Make sure that the change passes the CI checks.

This change is already covered by existing tests, such as (please describe tests).

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

  • [ ] Dependencies (add or upgrade a dependency)
  • [ ] The public API
  • [ ] The schema
  • [ ] The default values of configurations
  • [ ] The threading model
  • [ ] The binary protocol
  • [ ] The REST endpoints
  • [ ] The admin CLI options
  • [ ] The metrics
  • [ ] Anything that affects deployment

Documentation

  • [ ] doc
  • [ ] doc-required
  • [x] doc-not-needed
  • [ ] doc-complete

Matching PR in forked repository

PR in forked repository: https://github.com/thetumbled/pulsar/pull/62

thetumbled avatar Jul 24 '24 10:07 thetumbled

Could you also shorten the method name? The test testProducerDuplicationWithReceiptLostDedupEnabledAndUserControlSequenceIdMultiPartitioned even has 90 characters.

Currently, the checkstyle plugin does not work for tests somehow (see #23079). So there is no code style check for tests.

Done, thanks for review.

thetumbled avatar Jul 26 '24 08:07 thetumbled