apache
Support for user defined FFI functions
Description
Support transport UDF FFI library from client to Scheduler&Executor, and provoid interface load FFI into TaskDefinition's function_registry.
Relate to 2.2 Support for user defined functions
Describe the solution you'd like
1. Rust FFI
Datafusion already support loading rust FFI module, base on abi_stable
For rust ABI, we can send rust FFI library from client side to Scheduler&Executor, and load udf functiuons from the FFI library.
2. C/C++ FFI
For C/C++ FFI, we can provide a interface wrapper base on rust FFI by use cxx crate.
3. Wasm FFI
As @drauschenbach suggest, WASM is possible way to express UDF's in Rust. We can consider provide wasmtime udf for better secure and standards-compliant.
Describe alternatives you've considered
N/A
Additional context
N/A
I'm not sure what's your idea @westhide, udf support is not one size fits all solution, thus I'm not sure how it fits with current solution. We spent a lot of time removing code, I'd be careful adding code.
Can you please elaborate a bit more?
closing this issue. i personally believe ballista provides an easy way to plug your own FFI UDFs. there is no intention at the moment to make that part of ballista as we do not have bandwidth to maintain additional code.