spark icon indicating copy to clipboard operation
spark copied to clipboard

[SPARK-52032][SQL] Fix orc filter pushdown with null-safe equality operator

Open yuexing opened this issue 7 months ago • 1 comments

What changes were proposed in this pull request?

Background:As stated in SPARK-52032: When applying a filter using eqNullSafe (<=>, null-safe equality operator) to a DataFrame created from an ORC file, rows containing null values are incorrectly excluded, even though they should be retained.

Changes:1. Added a UT in OrcFilterSuite to reproduce the issue, 2. fix the logic in OrcFilter.

Why are the changes needed?

As stated above, it's an unexpected issue

Does this PR introduce any user-facing change?

No.

How was this patch tested?

UT

Was this patch authored or co-authored using generative AI tooling?

No

yuexing avatar May 19 '25 08:05 yuexing

Thank you @yuexing and @LuciferYang .

dongjoon-hyun avatar May 21 '25 00:05 dongjoon-hyun

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

github-actions[bot] avatar Aug 31 '25 00:08 github-actions[bot]