Andrew Lamb

My hope/plan is to figure out a good way to properly separate IO/Compute in the context of - https://github.com/apache/datafusion/pull/13424 I hope to have a chance to work on it more...

> @alamb, do you have any suggestions on the number of threads to allocate for cpu bound tasks vs IO bound ones relative to number of available cores? I will...

FWIW @adriangb reports success using this procedure here: - https://github.com/apache/datafusion/pull/13634

I took this idea a bit further (and wrote a bunch of tests) as part of this PR in DataFusion: - https://github.com/apache/datafusion/pull/14286

> > I took this idea a bit further (and wrote a bunch of tests) as part of this PR in DataFusion: > > > > * [Example for using...

> @alamb FYI pagination also relevant for local context I agree One way to do this might be to wrap the `SendableRecordBatchStream` in an object that buffers past results and...

💯

I will plan to try this after I have completed https://github.com/datafusion-contrib/datafusion-tui/issues/132

I have a PR ready: https://github.com/datafusion-contrib/datafusion-dft/pull/143 It went quite smootly

> Regarding the point on storing data - i wonder if the [CacheManager](https://github.com/apache/datafusion/blob/23f3a8f07f2ec907cbf1fd55afa75196d9e09278/datafusion/execution/src/cache/cache_manager.rs#L32) could be extended / used to serialize the files / file_stats caches. That is an excellent idea...