Caleb Van Dyke
Caleb Van Dyke
I think this in particular is just a batch source cause you can't really stream in a file. We do have a similar streaming source for GCS files but that...
The idea is you can have one processor that is an input into another processor. Right now this can kind of be accomplished by having the sink of one processor...
Things like avro and parquet that have a schema associated with them. If the user is writing a dataclass we should verify that the schema matches with the existing file...
Can look at starting with Google Cloud Spanner for this, but ideally we can figure out something generic enough to extend to any database that supports change streams.
Will likely just start with simple process time windowing (e.g. give me all messages in the last X seconds)