flink-connector-elasticsearch icon indicating copy to clipboard operation
flink-connector-elasticsearch copied to clipboard

[FLINK-35287] Builder builds NetworkConfig for Elasticsearch connector 8

Open liuml07 opened this issue 9 months ago • 1 comments

https://issues.apache.org/jira/browse/FLINK-35287

In FLINK-26088 we added support for ElasticSearch 8.0. It is based on Async sink API and does not use the base module flink-connector-elasticsearch-base. Regarding the config options (host, username, password, headers, ssl...), we pass all options from the builder to AsyncSink, and last to AsyncWriter. It is less flexible when we add new options and the constructors will get longer and multiple places may validate options unnecessarily. I think it's nice if we make the sink builder builds the NetworkConfig once, and pass it all the way to the writer. This is also how the base module for 6.x / 7.x is implemented.

In my recent work adding new options to the network config, this way works simpler. I'll port that work to 8.x

  1. builder builds NetworkConfig and pass that into sink and writer, instead of all specific config options
  2. other small fixes in the builder

liuml07 avatar May 03 '24 07:05 liuml07

@mtfelisb and @reswqa mind taking a look? Thanks!

liuml07 avatar May 03 '24 18:05 liuml07

@reswqa @MartijnVisser Could you help review? Thanks

liuml07 avatar May 21 '24 22:05 liuml07