flink
flink copied to clipboard
[FLINK-32622][table-planner] Optimize mini-batch assignment
What is the purpose of the change
The pull request detects unnecessary mini-batch assignments and ignores them in the query plan
Brief change log
- Search query plan if it contains aggregate or join operators
- If none of them exists in query plan, then we ignore the mini-batch assignment
- With that, we avoid generating useless events and causing performance issues
Verifying this change
This change added tests and can be verified as unit tests that I added as part of this PR
Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): no
- The public API, i.e., is any changed class annotated with
@Public(Evolving)
: no - The serializers: no
- The runtime per-record code paths (performance sensitive): no
- Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
- The S3 file system connector: no
Documentation
- Does this pull request introduce a new feature? no
CI report:
- d40e35c30a7fdcdd2aa99161dbc61bc3e07e0b61 Azure: SUCCESS
Bot commands
The @flinkbot bot supports the following commands:-
@flinkbot run azure
re-run the last Azure build
Hi @swuferhong , could you please review? Thanks!
Hi @JingGe @xuyangzhong thank you for your reviews. I am very sorry, somehow I haven't noticed your reviews in time. I will address your comments and update the PR ASAP.
Hi @JingGe @xuyangzhong thanks for your reviews. I updated the PR addressing your comments. Could you please check if you have further comments? I think the CI failure is not related to this PR
@flinkbot run azure
Thanks a lot @xuyangzhong for your review. I addressed your comments.
Hi @xuyangzhong thanks for the comment. I addressed your comments. Please, let me know if you agree.