flink icon indicating copy to clipboard operation
flink copied to clipboard

Published 20 hours ago • verified publisher icon apache

[FLINK-33980][core] Reorganize job configuration

Open JunRuiLee opened this issue 1 year ago • 10 comments

What is the purpose of the change

Reorganize job configuration

Brief change log

  1. Migrate configuration from non-ConfigOption objects to use ConfigOption.
  2. Adopt a single Configuration object to house all configurations.
  3. Create complex Java objects, such as RestartBackoffTimeStrategyFactory, CheckpointStorage, and StateBackend, directly from the configuration on the JM side.

Verifying this change

This change is already covered by existing tests.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): (yes / no)
  • The public API, i.e., is any changed class annotated with @Public(Evolving): (yes / no)
  • The serializers: (yes / no / don't know)
  • The runtime per-record code paths (performance sensitive): (yes / no / don't know)
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know)
  • The S3 file system connector: (yes / no / don't know)

Documentation

  • Does this pull request introduce a new feature? (yes / no)
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented)

JunRuiLee avatar Jan 04 '24 04:01 JunRuiLee

CI report:

  • 8ce6e37f513210a13c89a409a7689892f7a44d93 Azure: SUCCESS
Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

flinkbot avatar Jan 04 '24 04:01 flinkbot

@zhuzhurk Could you help to review this PR ? Thanks.

JunRuiLee avatar Jan 04 '24 05:01 JunRuiLee

Thanks @JunRuiLee for the contribution!

FLINK-33935 improves 2 options related to statebackend and checkpoint. It is merged,and it has a little conflicts with this PR, please address it in your free time, thanks~

Thanks @1996fanrui for the kind reminder. I have rebased this pull request and resolved the conflicts.

JunRuiLee avatar Jan 05 '24 05:01 JunRuiLee

Thanks @1996fanrui for the quick reviews, I've updated this pr, PTAL~

JunRuiLee avatar Jan 05 '24 09:01 JunRuiLee

Updated this PR with a rebase onto the master branch and resolved conflicts.

JunRuiLee avatar Jan 06 '24 10:01 JunRuiLee

Hi @JunRuiLee , would you mind rebaseing the master branch to fix the CI? thanks~

1996fanrui avatar Jan 09 '24 04:01 1996fanrui

Hi @JunRuiLee , would you mind rebaseing the master branch to fix the CI? thanks~

Hi @1996fanrui , I am currently waiting for the merge of https://github.com/apache/flink/pull/24046. Once that has been completed, I'll go ahead and remove the hotfix commit 6ba47965e234b6524f10dbe0d933e5e5dff6627, and then I will rebase onto the updated master branch. Thanks for your patience!

JunRuiLee avatar Jan 09 '24 06:01 JunRuiLee

Hi, @JunRuiLee, https://github.com/apache/flink/pull/24046 has been merged. You could rebase the master and go ahead.

masteryhx avatar Jan 10 '24 02:01 masteryhx

Thanks @masteryhx for the heads up, I've updated the PR following the merge.

JunRuiLee avatar Jan 10 '24 04:01 JunRuiLee

Thanks @JunRuiLee for the rebase!

Hi @zhuzhurk , would you like to review this PR as well? If no any comments, I will merge this PR next Monday.

Thanks for helping with the review! @1996fanrui I'd like to take another look before merging the changes.

zhuzhurk avatar Jan 11 '24 11:01 zhuzhurk

Thanks @zhuzhurk and @1996fanrui for your reviews, I've updated this pr accordingly, PTAL.

JunRuiLee avatar Jan 13 '24 09:01 JunRuiLee