spark
spark copied to clipboard
[SPARK-49940][CONNECT]Make 'limit n' in spark connect use executeCollect
What changes were proposed in this pull request?
Call executeCollect
when the DataFrame is CollectLimitExec
or CollectTailExec
.
Why are the changes needed?
For limit execution, SparkPlan.executeCollect
is more performant than SparkPlan.execute
.
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Existing tests
Was this patch authored or co-authored using generative AI tooling?
No