coral
coral copied to clipboard
[Coral-Incremental] Incremental plan generation for RelNode incremental rewrite
What changes are proposed in this pull request, and why are they necessary?
Introduction: The goal of this PR is to introduce Incremental Plan Generation for Coral-Incremental. This feature will enable the generation of multiple incremental plans for a given RelNode (a logical plan) based on the number of sub-queries that will be computed incrementally. The number of generated plans will match the number of sub-queries in a SQL query. Each logical plan in these generated plans will update the materialized view with its results.
For example, in a multi-table join, we can choose between batch and incremental execution stages. Consider the following three-table join:
LogicalProject#8
|
LogicalJoin#7
/ \
LogicalProject#4 TableScan#5
|
LogicalJoin#3
/ \
TableScan#0 TableScan#1
We could generate plans as follows:
Incremental: Both joins are executed incrementally.
Part-Batch, Part-Incremental: The first join is executed incrementally, and the second join is executed in batch mode.
Batch: Both joins are executed in batch mode.
Each generated plan will be represented as a List<RelNode>, where each RelNode indicates whether its corresponding sub-query will be executed incrementally or in batch mode.
Some important changes:
- New
RelNodeGenerationTransformer
to rewrite incremental format RelNodes by materializing the join subqueries, and combine them to generate different complete plans. - Added two helper classes within
RelNodeGenerationTransformer
:findJoinNeedsProject
: Converts allRelNodes
into a uniform format for future incremental plan generation.uniformFormat
: Ensures consistent formatting ofRelNodes
. -
convertRelPrev
method which could convert the tables of aRelNode
into '_prev' format, thus distinguishing from the original one.
How was this patch tested?
Unit test