spark icon indicating copy to clipboard operation
spark copied to clipboard

Published 20 hours ago • verified publisher icon apache

[SPARK-46741][SQL] Cache Table with CTE won't work

Open AngersZhuuuu opened this issue 1 year ago • 6 comments

What changes were proposed in this pull request?

Cache Table with CTE won't work, there are two reasons

  1. In the current code CTE in CacheTableAsSelect will be inlined
  2. CTERelation Ref and Def didn't handle the CTEId doCanonicalize issue

Cause the current case can't be matched.

Why are the changes needed?

Fix bug

Does this PR introduce any user-facing change?

Yea, Cache table with CTE can work after this pr

How was this patch tested?

Added UT

Was this patch authored or co-authored using generative AI tooling?

No

AngersZhuuuu avatar Jan 17 '24 09:01 AngersZhuuuu

ping @cloud-fan

AngersZhuuuu avatar Jan 17 '24 09:01 AngersZhuuuu

How about current? @cloud-fan

AngersZhuuuu avatar Jan 18 '24 02:01 AngersZhuuuu

ping @cloud-fan @yaooqinn

AngersZhuuuu avatar May 14 '24 02:05 AngersZhuuuu

@cloud-fan Changed follow the discussion

AngersZhuuuu avatar May 14 '24 09:05 AngersZhuuuu

ping @cloud-fan

AngersZhuuuu avatar May 16 '24 07:05 AngersZhuuuu

ping @cloud-fan @HyukjinKwon @dongjoon-hyun @viirya

AngersZhuuuu avatar Jul 02 '24 09:07 AngersZhuuuu

what was behaviour before? Would be great to show the result before/after

HyukjinKwon avatar Jul 03 '24 00:07 HyukjinKwon

what was behaviour before? Would be great to show the result before/after

For the query in cache.sql

EXPLAIN EXTENDED SELECT * FROM cache_nested_cte_table

before this pr, cached table cache_nested_cte_table won't match, will execute again, after this pr, it can match the InMemoryRelation

AngersZhuuuu avatar Jul 03 '24 02:07 AngersZhuuuu