iceberg-rust icon indicating copy to clipboard operation
iceberg-rust copied to clipboard

feat(datafusion): Support pushdown more datafusion exprs to Iceberg

Open FANNG1 opened this issue 1 year ago • 1 comments

fixes: #644

DataFusion expressions and Iceberg predicates do not have a one-to-one correspondence.

  1. Columns and literals are represented as Expr in DataFusion, whereas they are merely components of predicates in Iceberg.
  2. And and Or are operators in DataFusion, but they are treated as predicates in Iceberg.

This disparity complicates the transformation process when using TreeNode. To address this, I also refactored the code to simplify it.

FANNG1 avatar Sep 26 '24 01:09 FANNG1

@liurenjie1024 @Xuanwo @sdd @a-agmon please help to review when you are free, thanks

FANNG1 avatar Sep 29 '24 00:09 FANNG1

thanks @sdd @a-agmon @Xuanwo for the reviewing.

FANNG1 avatar Oct 12 '24 06:10 FANNG1