flink icon indicating copy to clipboard operation
flink copied to clipboard

[FLINK-32622][table-planner] Optimize mini-batch assignment

Open jeyhunkarimov opened this issue 1 year ago • 3 comments

What is the purpose of the change

The pull request detects unnecessary mini-batch assignments and ignores them in the query plan

Brief change log

  • Search query plan if it contains aggregate or join operators
  • If none of them exists in query plan, then we ignore the mini-batch assignment
  • With that, we avoid generating useless events and causing performance issues

Verifying this change

This change added tests and can be verified as unit tests that I added as part of this PR

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no

jeyhunkarimov avatar Sep 26 '23 20:09 jeyhunkarimov

CI report:

  • d40e35c30a7fdcdd2aa99161dbc61bc3e07e0b61 Azure: SUCCESS
Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

flinkbot avatar Sep 26 '23 20:09 flinkbot

Hi @swuferhong , could you please review? Thanks!

jeyhunkarimov avatar Sep 27 '23 15:09 jeyhunkarimov

Hi @JingGe @xuyangzhong thank you for your reviews. I am very sorry, somehow I haven't noticed your reviews in time. I will address your comments and update the PR ASAP.

jeyhunkarimov avatar Jan 08 '24 20:01 jeyhunkarimov

Hi @JingGe @xuyangzhong thanks for your reviews. I updated the PR addressing your comments. Could you please check if you have further comments? I think the CI failure is not related to this PR

jeyhunkarimov avatar Jan 14 '24 16:01 jeyhunkarimov

@flinkbot run azure

jeyhunkarimov avatar Jan 18 '24 11:01 jeyhunkarimov

Thanks a lot @xuyangzhong for your review. I addressed your comments.

jeyhunkarimov avatar Apr 17 '24 16:04 jeyhunkarimov

Hi @xuyangzhong thanks for the comment. I addressed your comments. Please, let me know if you agree.

jeyhunkarimov avatar Apr 26 '24 06:04 jeyhunkarimov