buildflow
buildflow copied to clipboard
launchflow
BuildFlow, is an open source framework for building large scale systems using Python. All you need to do is describe where your input is coming from and where your output should be written, and BuildF...
if you have low throughput with massive fan out it will actually be better to use loads jobs, we should support this option.
Right now we just fail when we start but things still attempt to start. Ideally this would kill the pipeline immediately.
Some more file types: JSON, Avro, CSV
I think this in particular is just a batch source cause you can't really stream in a file. We do have a similar streaming source for GCS files but that...
The idea is you can have one processor that is an input into another processor. Right now this can kind of be accomplished by having the sink of one processor...
Things like avro and parquet that have a schema associated with them. If the user is writing a dataclass we should verify that the schema matches with the existing file...
Can look at starting with Google Cloud Spanner for this, but ideally we can figure out something generic enough to extend to any database that supports change streams.
Will likely just start with simple process time windowing (e.g. give me all messages in the last X seconds)
Metadata
Owner
Metadata
BuildFlow, is an open source framework for building large scale systems using Python. All you need to do is describe where your input is coming from and where your output should be written, and BuildF...