flink-connector-elasticsearch icon indicating copy to clipboard operation
flink-connector-elasticsearch copied to clipboard

[FLINK-36143][ES6][ES7] Intro retry-on-conflict param to resolve Sink ES occurred "version conflict"

Open leosanqing opened this issue 1 year ago • 5 comments

When I used ES connector Write update datas, it occurred "version conflict", like this " org.elasticsearch.index.engine.VersionConflictEngineException: [project][1140860]: version conflict, current version [5] is different than the one provided [4] " When we set the conf failure-handler = ignore, It lead to lost data, and if set the conf failure-handler = fail, job will restore. this cost is too heavy. maybe second try it will success.

so I intro this conf param, this will make job failed cost smaller.

leosanqing avatar Aug 23 '24 09:08 leosanqing

Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html)

boring-cyborg[bot] avatar Aug 23 '24 09:08 boring-cyborg[bot]

@schulzp @alpinegizmo @aljoscha @rmetzger Has any body to review my code? thk ❤️❤️

leosanqing avatar Aug 27 '24 09:08 leosanqing

Thanks for your advices,and I fixed this. Has anybody can review my code?

Thank you for your patch! I am not an official maintainer. Your changes look reasonable, however, they lack test coverage. One other thing I noticed: The newly introduced constant RETRY_ON_CONFLICT_NUM does not follow the naming convention of other config options in ElasticsearchConnectorOptions: Most of them appear to end on [...]_OPTION but do not have the type-indicating suffix. Renaming it to sink.retries-on-conflict could eliminate the confusion wether this is a boolean flag or a number.

leosanqing avatar Aug 30 '24 06:08 leosanqing

@reswqa @mtfelisb hi guys,could you review my pr? thx

leosanqing avatar Sep 06 '24 06:09 leosanqing

@rmetzger hi, thx for your triggering. I see there has some error when run IT case. And there also has same error in before PR . Is there some problems in project? The leatest merged pr doesn't have this step, latest un-merged PRs both have this error.

leosanqing avatar Sep 06 '24 07:09 leosanqing