spark icon indicating copy to clipboard operation
spark copied to clipboard

[SPARK-46741][SQL] Cache Table with CTE won't work

Open AngersZhuuuu opened this issue 3 weeks ago • 2 comments

What changes were proposed in this pull request?

Reopen https://github.com/apache/spark/pull/44767 Cache Table with CTE won't work, there are two reasons

  1. In the current code CTE in CacheTableAsSelect will be inlined
  2. CTERelation Ref and Def didn't handle the CTEId doCanonicalize issue Cause the current case can't be matched.

Why are the changes needed?

Fix Bug

Does this PR introduce any user-facing change?

Yea, Cache table with CTE can work after this pr

For added cache.sql final query EXPLAIN EXTENDED SELECT * FROM cache_nested_cte_table;

Before this pr, the plan as below, cache won't work. 截屏2025-12-05 11 22 05

After this pr 截屏2025-12-05 11 32 38

How was this patch tested?

Added UT

Was this patch authored or co-authored using generative AI tooling?

No

AngersZhuuuu avatar Dec 05 '25 03:12 AngersZhuuuu

@cloud-fan Could you take a look again? in https://github.com/apache/spark/pull/44767 you approved but not merge this pr. This bug really impact product job's performance

AngersZhuuuu avatar Dec 05 '25 03:12 AngersZhuuuu

ping @cloud-fan

AngersZhuuuu avatar Dec 08 '25 03:12 AngersZhuuuu

thanks, merging to master/4.1!

cloud-fan avatar Dec 18 '25 07:12 cloud-fan