apache
Make the configuration for `StreamTable` more generic to support more stream sources
Is your feature request related to a problem or challenge?
I am working on a websocket TableProvider and initially I went about creating my own TableProvider but then after reviewing the StreamTable and StreamConfig it seems it was made with the potential to be generic at some point.
Ref: "Currently only reading from / appending to a single file in-place is supported, but other stream sources and sinks may be added in future."
So I decided to try making a more generic interface for this. As a first step I created the new interface and ported the existing implementation to use this (I will have a PR up shortly). I havent finished cleaning up yet but i think the API is in good enough shape to start getting feedback.
The TLDR of the new interface is that there is a new trait StreamSource which provides the Schema and RecordBatchReader and RecordBatchWriter. Then there are implementations like FileStreamSource or WebsocketStreamSource.
Assuming this is going in the right direction I can cleanup more, get this merged, and then follow on with an implementation of WebsocketStreamSource.
Describe the solution you'd like
No response
Describe alternatives you've considered
No response
Additional context
No response
