flink
flink copied to clipboard
[FLINK-28820][Connector/Pulsar] Improve the writing performance for PulsarSink
What is the purpose of the change
PulsarSink is a bit slow when we use the At-Least-Once
or Exactly-Once
. This is because PulsarSink uses MallboxExecutor
to write messages. A lot of useless context switches and locks acquire happened when using this writing policy. We should drop MallboxExecutor
and send messages directly for better performance.
Brief change log
- Deprecated useless
PulsarSinkOptions.PULSAR_MAX_PENDING_MESSAGES_ON_PARALLELISM
option. - Remove the use of
MallboxExecutor
inPulsarWriter
.
Verifying this change
This change is already covered by existing tests, such as PulsarWriterTest.
Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
@Public(Evolving)
: (no) - The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not applicable)
CI report:
- 250a287ecc41c0049387d73c3a27f3fb41af3113 Azure: FAILURE
Bot commands
The @flinkbot bot supports the following commands:-
@flinkbot run azure
re-run the last Azure build