datafusion icon indicating copy to clipboard operation
datafusion copied to clipboard

Add `async` UDF example

Open marshauf opened this issue 2 years ago • 5 comments

Which issue does this PR close?

Closes #6518.

Rationale for this change

This change provides an example on how to replace a ScalarUDF node with a user defined extension node. The node calls an async function on execution.

What changes are included in this PR?

It only adds another example which can be run with cargo run --example async_udf.

Are these changes tested?

I ran cargo run --example async_udf which ran the example to completion without an error. Example has no external dependencies towards its environment.

Are there any user-facing changes?

No user facing changes.

TODO

At the moment the optimizer rule replaces the complete Projection. I would like to just replace the Expr::ScalarUDF with a Expr::ScalarSubquery or something similar which allows me to host the user defined extension node. An example implementation is rewrite_expr, but Expr::ScalarSubquery doesn't support LogicalPlan::Extension as a subquery. A function returns an error at https://github.com/apache/arrow-datafusion/blob/main/datafusion/optimizer/src/scalar_subquery_to_join.rs#L219-L219 because https://github.com/apache/arrow-datafusion/blob/main/datafusion/expr/src/logical_plan/plan.rs#L411-L411 returns None for LogicalPlan::Extension.

marshauf avatar Jun 18 '23 10:06 marshauf

@alamb can you help me with the task in the TODO section, please. I would like to keep the Projection node and just replace the ScalarUDF Expr with something which can host an Extension. I hoped a SubQuery would work, but it seems to not work/be supported.

marshauf avatar Jun 20 '23 15:06 marshauf

@alamb can you help me with the task in the TODO section, please. I would like to keep the Projection node and just replace the ScalarUDF Expr with something which can host an Extension. I hoped a SubQuery would work, but it seems to not work/be supported.

Yes, I will be happy to -- I may not have time until later in this week however.

alamb avatar Jun 20 '23 21:06 alamb

I have not forgotten about this -- I hope/plan to spend time on the weekend

alamb avatar Jun 23 '23 15:06 alamb

Marking as draft to signify it is not waiting on review

alamb avatar Aug 16 '23 18:08 alamb

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

github-actions[bot] avatar May 05 '24 01:05 github-actions[bot]