Report splits to DataFusion as partitions

[gatesn]

[gatesn]

@robert3005 do you think we want to use this + InlineExecutor?

[robert3005]

I think if you do this you might still want a threaded runtime (my point of reference is that in spark parquet still spins up a forkjoinpool per task), just that each one of those partitions will have fewer row splits. Fwiw this is mostly useful in case of failures since I assume datafusion will retry only failing partitions.

[gatesn]

The benefit of this is that DataFusion can choose to parallelise the rest of the execution plan, whereas in the current model, we do use all threads of the runtime to scan a file, but DataFusion sees a serialized sequence of record batches when in reality the ordering often doesn't matter at all.