Andy Grove
Andy Grove
@Dandandan Any suggestions on tracking this down?
I built Ballista from [this PR](https://github.com/apache/arrow-ballista/pull/927) with `cargo build --release`. My client is at https://github.com/sql-benchmarks/sqlbench-runners/pull/32 and has these dependencies: ```toml [dependencies] ballista = { git = "https://github.com/andygrove/arrow-ballista", branch="df-34" } datafusion...
> Is the branch compiling already? Yes, it was just the tests that weren't compiling in CI. I pushed a fix. I ran a cargo clean locally on ballista and...
> Does enabling it in the sqlbenchrunner as well by explicitly adding `arrow` dependency with `ipc_compression` work? Yes, that fixed it, thanks. I can create a PR to add this...
Yes, it could be supported, at least to some degree. I looked at the [docs](https://spark.apache.org/docs/latest/spark-connect-overview.html), and it looks like Spark Connect is essentially a gRPC service that accepts protobuf describing...
@thinkharderdev @yahoNanJing @mingmwang @avantgardnerio Let me know what you think. If this seems like a good idea, I may have some time later this month to try and implement it....
I ran into some challenges with the proposed design, which I have documented in the Google doc (near the end). Feedback welcome.
This sounds great @yahoNanJing. I would also like to hear if @yjshen has an opinion on this.
@Dandandan just fyi, here is current status
Whatever we do here, it would be good to consider the design in DataFusion as well since we need to make the two work together.