Dan Harris

How would you support all-at-once scheduling in this setup for streaming execution?

> > How would you support all-at-once scheduling in this setup for streaming execution? > > Do you mean long running tasks for streaming case? Would you submit hundreds of...

To be transparent, my team is building a query engine which is sensitive to time-to-first-result latency so we are very interested in fully streaming execution (and hoping to upstream as...

cc @andygrove @yahoNanJing @realno @Ted-Jiang

> I am curious when you said fully streaming execution did you mean like Flink? Exactly, the current execution model is basically Flink Batch execution, but what we ultimately want...

Great, thanks for the feedback everyone! I would summarize the key points as: 1. Based on this and previous discussions, the goal is not to create yet another general "distributed...

I agree with @yahoNanJing on his answers above. The only thing I would add is that with respect to deployment we should be able to support different auto-scaling behaviors for...

> I also have some thoughts about a unified execution engine, welcome to take a look and comment: > > https://www.notion.so/liurenjie1024/A-Cloud-Native-Universal-Execution-Engine-7903dd9eeea143c48049631a2d1cb845 > > cc @andygrove @mingmwang Thanks @liurenjie1024. I recently...

> > I also have some thoughts about a unified execution engine, welcome to take a look and comment: > > https://www.notion.so/liurenjie1024/A-Cloud-Native-Universal-Execution-Engine-7903dd9eeea143c48049631a2d1cb845 > > cc @andygrove @mingmwang > > How...

This PR would support this use case: https://github.com/apache/arrow-datafusion/pull/1677