flink icon indicating copy to clipboard operation
flink copied to clipboard

[FLINK-24024][table-planner] support session window tvf in plan

Open xuyangzhong opened this issue 2 years ago • 1 comments

What is the purpose of the change

This pull request aims to introduce the support of the session window tvf in the plan. After this pr, users can use the syntax with session window tvf with SQL.

Note: this pr is not related to table-runtime. And the session window tvf this pr introduced only has the basic features like those described in https://issues.apache.org/jira/browse/FLINK-23544 .

Some flaws will be fixed later because this pr is a little big such as removing duplicated Exchange node, supporting any order of group keys and etc.

Brief change log

  • adapt the requirement about ptf in calcite: full the table characteristic of the session window tvf.
  • adapt some rules about introducing session window tvf.
  • add ut and it cases for it.

Verifying this change

Some ut and it cases are added to verify this pr.

Does this pull request potentially affect one of the following parts:

  • Dependencies (does it add or upgrade a dependency): no
  • The public API, i.e., is any changed class annotated with @Public(Evolving): no
  • The serializers: no
  • The runtime per-record code paths (performance sensitive): no
  • Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
  • The S3 file system connector: no

Documentation

  • Does this pull request introduce a new feature? no
  • If yes, how is the feature documented? no

xuyangzhong avatar Oct 11 '23 03:10 xuyangzhong

CI report:

  • a093035139a2545e6fa0db1975b6040c002edd76 Azure: PENDING
Bot commands The @flinkbot bot supports the following commands:
  • @flinkbot run azure re-run the last Azure build

flinkbot avatar Oct 11 '23 03:10 flinkbot

Hi, @LadyForest @snuyanzin . Thank you for taking the time to review. I have updated the PR. If you have time, please check again.

xuyangzhong avatar Jan 16 '24 11:01 xuyangzhong

I'm not splitting this pr to multi parts because the most of this changes are new filed generated by json plan test(almost 2000 lines). However, I have splited the pr supporting session window tvf into two parts(one is this pr, another is in table-runtime in FLINK-34048). Actually there are relatively few changes to the planner. Further more, it is necessary to separately add tests for the session window plan at different stages. This may not be well unified with the tests of other window tvfs.

xuyangzhong avatar Jan 16 '24 11:01 xuyangzhong

I would vote for having order by support as welll within this PR (assuming that there is no blockers for that). WDYT?

snuyanzin avatar Jan 18 '24 23:01 snuyanzin

Hi, @snuyanzin . Regarding this PR, my original intention was to implement it according to the existing FLIP and not to introduce features that haven't been discussed. IMO, although Calcite provides support for ORDER BY on SET SEMANTICS TABLE, supporting order by for session window needs to be further discussed in a separate JIRA. As far as this PR is concerned, I have reserved the order by field in RexSetSemanticsTableCall. If it is discussed in a subsequent separate JIRA thread that it is necessary to add order by syntax to the session window tvf, we can also quickly support it. WDYT?

xuyangzhong avatar Jan 19 '24 01:01 xuyangzhong

@flinkbot run azure

xuyangzhong avatar Jan 24 '24 05:01 xuyangzhong

@flinkbot run azure

xuyangzhong avatar Jan 24 '24 08:01 xuyangzhong

@flinkbot run azure

xuyangzhong avatar Jan 24 '24 09:01 xuyangzhong