delta icon indicating copy to clipboard operation
delta copied to clipboard

[SPARK] Fix infinite loops when has foldable dataFilter in data skipping

Open zml1206 opened this issue 1 year ago • 1 comments

Which Delta project/connector is this regarding?

  • [x] Spark
  • [ ] Standalone
  • [ ] Flink
  • [ ] Kernel
  • [ ] Other (fill in here)

Description

This PR to fix #2904. I have submitted PR to spark for optimize predicate after push extra predicate through join(https://github.com/apache/spark/pull/46085), but this is spark's plan optimizer, we can't guarantee whether predicates like 1=1 will be generated in other situations.

How was this patch tested?

New unit test.

Does this PR introduce any user-facing changes?

No.

zml1206 avatar Apr 17 '24 09:04 zml1206

cc @scottsand-db Do you have time to take a look? Thank you.

zml1206 avatar Apr 24 '24 07:04 zml1206

Resolved by #3059

zml1206 avatar May 09 '24 09:05 zml1206