apache
[FLINK-36066][runtime] Introducing the AdaptiveGraphManager component
What is the purpose of the change
To support the incremental generation of JobGraph and update the StreamGraph based on runtime information, we plan to introduce the AdaptiveGraphManager component, which will implement the AdaptiveGraphGenerator interface. The AdaptiveGraphGenerator interface is responsible for:
- Generating the JobGraph
- Responding to upstream job vertex finished events, generating JobVertex, and updating it to the JobGraph
- Providing StreamGraphContext to allow StreamGraphOptimizer to add, delete, and modify StreamNode and StreamEdge
Brief change log
- Implement the AdaptiveGraphGenerator interface
- Implement the StreamGraphContext interface
- Some UT cases
Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (no)
- The public API, i.e., is any changed class annotated with
@Public(Evolving): (no) - The serializers: (no)
- The runtime per-record code paths (performance sensitive): (no)
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no)
- The S3 file system connector: (no)
Documentation
- Does this pull request introduce a new feature? (no)
- If yes, how is the feature documented? (not documented)
CI report:
- 8dce314977a27765727a468987f96f83fcd41caf UNKNOWN
- bb315a49b46c8a9c56056a2f989fc6172fc4ad73 Azure: SUCCESS
Bot commands
The @flinkbot bot supports the following commands:@flinkbot run azurere-run the last Azure build
Thank @noorall for the update. Such a complex refactoring is no easy task. I've left some comments, please take a look
Thanks for your review. The PR has been updated according to your comments.
Thanks for addressing all the comments above. @noorall I have a few comments regarding the first 2 commits.
Thanks for your review. The PR has been updated according to your comments.