datafusion-ballista
datafusion-ballista copied to clipboard
apache
Apache Arrow Ballista Distributed Query Engine
Tracking tasks with DataFusion issues which ballista benefits from: ## DataFusion
Following the completion of #1068, it's time to propose the next steps for Ballista. In the short term, I would like to focus on the following areas: - **Improving test...
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** Improve shuffle reading, with https://github.com/apache/arrow-rs/pull/7120 merged consider using it for shuffle files....
## Description Support transport UDF FFI library from client to Scheduler&Executor, and provoid interface load FFI into `TaskDefinition`'s `function_registry`. Relate to [2.2 Support for user defined functions](https://github.com/apache/datafusion-ballista/issues/1068#issue-2581806623) ## Describe the...
**Describe the bug** Ballista client keep blocking when prepare_task_definition or prepare_multi_task_definition fail Relate to [#1212](https://github.com/apache/datafusion-ballista/pull/1212#issue-2933515017) **To Reproduce** Steps to reproduce the behavior: Run the `should_support_json_source` test in [#1212](https://github.com/apache/datafusion-ballista/pull/1212#issue-2933515017) ```rust //...
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** I would like to be able to `register external table stored as...
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** A new feature enhancement **Describe the solution you'd like** Current Ballista’s shuffle...
First of all, I'm not expert in rust-python (pyo3) integration, if I've done/said something stupid, my apologies. Current implementation of (py)ballista has limitation when it comes to `DataFrame` operations. following...
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** Create the PyPi release process for [Ballista Python](https://pypi.org/project/ballista/). (This section helps Arrow...
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.** We need to identify gaps between ballista and datafusion and address them,...