flink icon indicating copy to clipboard operation
flink copied to clipboard

[FLINK-28820][Connector/Pulsar] Improve the writing performance for PulsarSink

Open syhily opened this issue 1 year ago • 1 comments

What is the purpose of the change

PulsarSink is a bit slow when we use the At-Least-Once or Exactly-Once. This is because PulsarSink uses MallboxExecutor to write messages. A lot of useless context switches and locks acquire happened when using this writing policy. We should drop MallboxExecutor and send messages directly for better performance.

Brief change log

  • Deprecated useless PulsarSinkOptions.PULSAR_MAX_PENDING_MESSAGES_ON_PARALLELISM option.
  • Remove the use of MallboxExecutor in PulsarWriter.

Verifying this change

This change is already covered by existing tests, such as PulsarWriterTest.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (no)
  • The serializers: (no)
  • The runtime per-record code paths (performance sensitive): (no)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
  • The S3 file system connector: (no)

Documentation

  • Does this pull request introduce a new feature? (no)
  • If yes, how is the feature documented? (not applicable)

syhily avatar Aug 08 '22 12:08 syhily

CI report:

  • 250a287ecc41c0049387d73c3a27f3fb41af3113 Azure: FAILURE
Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

flinkbot avatar Aug 08 '22 13:08 flinkbot