Support Reading Substrait plans

[universalmind303]

Description

supporting substrait plans opens up a lot of possibilities of easily integrating with external tools (such as polars).

[eitsupi]

It would be great if this were supported, as it seems that GlareDB is already compiled with datafusion-substrait built in.

[tychoish]

@eitsupi what did you have in mind for using this? We never prioritized this because we didn't have a use case or a thing that could generate substrait plans that we were aware of people using.

I definitely think that it'd be cool, but I'd like to collect some potential use cases and tools that we could integrate with before working on this directly.

[eitsupi]

Fair point. apache/arrow#37504 is what I would like to see implemented.

There are dedicated integrations for the Apache Arrow and DuckDB packages in Python and R^1, but it would be great if this could be generalized to perform pushdown between the various packages. Of course it will take a while for this to happen, so I don't know if it would be worth implementing right away.

[tychoish]

Some of this sounds a lot like ADBC.

Also, to be clear, parts of this makes a lot of sense, particularly being able to take data formats and objects from the integration packages and pass the results of a GlareDB query to a DuckDB query (say). It should be possible to convert the arrow batches in such a way that this would work pretty well (or well enough,) and is something that could be extended, would not require a lot of work, and makes sense.

Pushing predicates down, between packages makes slightly less sense, or at least in most cases you'd want the query engine to connect to (and track push downs) the underlying datasource, rather than doing the translation this in the bindings.

I can imagine specific integration between query engines relying on substrait or another protocol, and if we were to implement that, using substrait when it was appropriate, could be great (and that can be easily accomplished, but I don't know (given the existing datafusion support for substrait), if there is anything particularly generic. to be done.