spark icon indicating copy to clipboard operation
spark copied to clipboard

[SPARK-46707][SQL][FOLLOWUP] Push down throwable predicate through aggregates

Open zml1206 opened this issue 1 year ago • 2 comments

What changes were proposed in this pull request?

Push down throwable predicate through aggregates and add ut for "can't push down nondeterministic filter through aggregate".

Why are the changes needed?

If we can push down a filter through Aggregate, it means the filter only references the grouping keys. The Aggregate operator can't reduce grouping keys so the filter won't see any new data after pushing down. So push down throwable filter through aggregate does not affect exception thrown.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

UT

Was this patch authored or co-authored using generative AI tooling?

No.

zml1206 avatar Feb 01 '24 03:02 zml1206

cc @cloud-fan

zml1206 avatar Feb 01 '24 09:02 zml1206

cc @kelvinjian-db

cloud-fan avatar Feb 02 '24 06:02 cloud-fan

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

github-actions[bot] avatar May 13 '24 00:05 github-actions[bot]

Is there still any problem with this PR, Can it be merged? Can you help take a look @cloud-fan

zml1206 avatar May 13 '24 02:05 zml1206

thanks, merging to master!

cloud-fan avatar May 14 '24 07:05 cloud-fan

Thank you all for review. @cloud-fan @kelvinjian-db @beliefer

zml1206 avatar May 14 '24 07:05 zml1206