flink-connector-elasticsearch icon indicating copy to clipboard operation
flink-connector-elasticsearch copied to clipboard

Published 20 hours ago • verified publisher icon apache

[FLINK-38429] Reuse OperationSerializer in Elasticsearch 8 connector to improve perf

Open beliefer opened this issue 2 months ago • 1 comments

This PR proposes to improve perf for Elasticsearch 8 connector. Currently, Elasticsearch 8 connector create a new OperationSerializer for each Operation. A large amount of OperationSerializer's instances brings big overhead.

There is a benchmark cases.

  1. Flink application with single parallelism. The Kafka Topic consumption rate show below.
Before this PR After this PR
1.5 K strips/s(min) 2.3 K strips/s(min)
2.0 K strips/s(max) 2.9 K strips/s(max)
single
  1. Flink application with 32 parallelisms. The Kafka Topic consumption rate show below.
Before this PR After this PR
58.3 K strips/s(min) 70.3 K strips/s(min)
60.0 K strips/s(max) 74.5 K strips/s(max)
32

beliefer avatar Sep 25 '25 03:09 beliefer