coral icon indicating copy to clipboard operation
coral copied to clipboard

[Coral-Incremental] Incremental plan generation for RelNode incremental rewrite

Open yyy1000 opened this issue 6 months ago • 1 comments

What changes are proposed in this pull request, and why are they necessary?

Introduction: The goal of this PR is to introduce Incremental Plan Generation for Coral-Incremental. This feature will enable the generation of multiple incremental plans for a given RelNode (a logical plan) based on the number of sub-queries that will be computed incrementally. The number of generated plans will match the number of sub-queries in a SQL query. Each logical plan in these generated plans will update the materialized view with its results.

For example, in a multi-table join, we can choose between batch and incremental execution stages. Consider the following three-table join:

               LogicalProject#8
                    |
               LogicalJoin#7
               /        \
       LogicalProject#4   TableScan#5
               |
        LogicalJoin#3
             /   \
   TableScan#0  TableScan#1

We could generate plans as follows:

Incremental: Both joins are executed incrementally.
Part-Batch, Part-Incremental: The first join is executed incrementally, and the second join is executed in batch mode.
Batch: Both joins are executed in batch mode.

Each generated plan will be represented as a List<RelNode>, where each RelNode indicates whether its corresponding sub-query will be executed incrementally or in batch mode.

Some important changes:

  1. New RelNodeGenerationTransformer to rewrite incremental format RelNodes by materializing the join subqueries, and combine them to generate different complete plans.
  2. Added two helper classes within RelNodeGenerationTransformer: findJoinNeedsProject: Converts all RelNodes into a uniform format for future incremental plan generation. uniformFormat: Ensures consistent formatting of RelNodes.
  3. convertRelPrev method which could convert the tables of a RelNode into '_prev' format, thus distinguishing from the original one.

How was this patch tested?

Unit test

yyy1000 avatar Jul 29 '24 21:07 yyy1000