spark
spark copied to clipboard
[SPARK-49982][SQL] Fix negative caching in InMemoryRelation
What changes were proposed in this pull request?
Re-cache AQE plan upon failure.
Why are the changes needed?
When we use a cached an AQE plan, it will do cachedPlan.execute to build the RDD, which will execute all AQE stages except the result stage. If any of them failed, the failure will be cached by lazy RDD val. So the next time when we reuse that cached plan (even by a totally irrelevant caller) it will fail immediately.
We need to re-cache the AQE plan upon failure.
Does this PR introduce any user-facing change?
NO
How was this patch tested?
new UT
Was this patch authored or co-authored using generative AI tooling?
NO
cc @maryannxue
thanks, merging to master!