spark icon indicating copy to clipboard operation
spark copied to clipboard

[SPARK-49982][SQL] Fix negative caching in InMemoryRelation

Open liuzqt opened this issue 1 year ago • 1 comments

What changes were proposed in this pull request?

Re-cache AQE plan upon failure.

Why are the changes needed?

When we use a cached an AQE plan, it will do cachedPlan.execute to build the RDD, which will execute all AQE stages except the result stage. If any of them failed, the failure will be cached by lazy RDD val. So the next time when we reuse that cached plan (even by a totally irrelevant caller) it will fail immediately.

We need to re-cache the AQE plan upon failure.

Does this PR introduce any user-facing change?

NO

How was this patch tested?

new UT

Was this patch authored or co-authored using generative AI tooling?

NO

liuzqt avatar Oct 15 '24 23:10 liuzqt

cc @maryannxue

HyukjinKwon avatar Oct 17 '24 01:10 HyukjinKwon

thanks, merging to master!

cloud-fan avatar Oct 24 '24 02:10 cloud-fan