factotum
factotum copied to clipboard
snowplow
A system to programmatically run data pipelines
Imagine this Factfile step: ``` json TODO ``` And imagine this is the referenced SQL Runner playbook: ``` yaml TODO ``` Then Factotum would pre-process this into the following inner...
Configurable frequency but default to 5 minutes. This is useful for the Factotum UI because if you have a very long-running job node (like a whole EMR jobflow), then there's...
As a webhook, we don't have a device sent timestamp, so we need an updateTime (or equivalent name) sent with each event.
The idea being that some information might not be suitable for transmission.
Hi, Often if a job is composed of many stages it is necessary to perform clean up activities if a certain stage fails. Is there some way to specify the...
Just an idea at this stage - a FSM is a particular flavor of DAG; could we implement an FSM behavior for Factotum? This could end up being an open-source...
@ninjabear - I noticed Factotum creates .factotum directory with factotum.log in it in the location from which you run the Factotum and not where the actual executable is located. That...
This is the idea of using a poll-based trigger to kick off a factfile. This is a potential building block for reactive batch. The process would be as follows: *...
The naming for this is a bit difficult to follow. We need to come up with a good, clear name for a "no operation" and what it means
