spark icon indicating copy to clipboard operation
spark copied to clipboard

[SPARK-48697][SQL] Add collation aware string filters

Open stefankandic opened this issue 8 months ago • 2 comments

What changes were proposed in this pull request?

Adding a new classes of filters which are collation aware.

Why are the changes needed?

#46760 Added the logic of predicate widening for collated column references, but this would completely change the filters and if the original expression did not get evaluated by spark later we could end up with wrong results. Also, data sources would never be able to actually support these filters and they would just see them as AlwaysTrue.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

New UTs.

Was this patch authored or co-authored using generative AI tooling?

No.

stefankandic avatar Jun 21 '24 16:06 stefankandic