delta
delta copied to clipboard
[SPARK] Fix infinite loops when has foldable dataFilter in data skipping
Which Delta project/connector is this regarding?
- [x] Spark
- [ ] Standalone
- [ ] Flink
- [ ] Kernel
- [ ] Other (fill in here)
Description
This PR to fix #2904.
I have submitted PR to spark for optimize predicate after push extra predicate through join(https://github.com/apache/spark/pull/46085), but this is spark's plan optimizer, we can't guarantee whether predicates like 1=1 will be generated in other situations.
How was this patch tested?
New unit test.
Does this PR introduce any user-facing changes?
No.
cc @scottsand-db Do you have time to take a look? Thank you.
Resolved by #3059