datafusion icon indicating copy to clipboard operation
datafusion copied to clipboard
verified publisher icon apache

Support `IS NOT NULL` predicates in PruningPredicate

Open alamb opened this issue 1 year ago • 3 comments

Is your feature request related to a problem or challenge?

the PruningPredicate currently does not handle IS NOT NULL predicates

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

@appletreeisyellow added an initial implementation in https://github.com/apache/arrow-datafusion/pull/9208 but I believe it didn't work in all cases, as described in https://github.com/apache/arrow-datafusion/issues/9230

alamb avatar Feb 14 '24 21:02 alamb

take

guojidan avatar Feb 18 '24 02:02 guojidan

Perhaps @appletreeisyellow is more suitable for handling this issue😄

guojidan avatar Feb 18 '24 03:02 guojidan

One thing I should have noted in the description here is that I think implementing IS NOT NULL will require a new API to PruningStatistics (the total_row_count) . Conveniently, @appletreeisyellow is adding this API as part of https://github.com/apache/arrow-datafusion/issues/9171 I think

So in other words, this ticket is likely blocked until #9171 is done

BTW there is a bunch more backstory on https://github.com/apache/arrow-datafusion/issues/9230

alamb avatar Feb 19 '24 06:02 alamb

I believe that @Ted-Jiang implemented this in https://github.com/apache/datafusion/pull/9989 + https://github.com/apache/datafusion/pull/10051 so closing this issue

alamb avatar Apr 22 '24 11:04 alamb