factotum
factotum copied to clipboard
snowplow
A system to programmatically run data pipelines
This is the simpler case of #39 - instead of having to execute a shell command to retrieve an inner DAG, we just retrieve it from a factfile on local...
We need this so we can start migrating makefiles over to Factotum. ``` bash $ factotum job make2factotum FILE ``` where `FILE` is a minimalistic Makefile, containing only the following...
After having a think about this, there's a couple of options for how to provision a job: 1) include the provisioning as a prerequisite task in the DAG - this...
Working with rustc-serialize is a bit tedious, and has a few limitations. We should move to [serde](https://github.com/serde-rs/serde) once the good codegen support lands in Rust stable.
The idea builds on #45, but instead of manually specifying boxes, you would just let all boxes attempt to run all jobs they find in their cron, and there would...
This functionality is in support of: - [EmrEtlRunner: placeholder for "Factotum inside"](https://github.com/snowplow/snowplow/issues/2577#issuecomment-208261179) - [Placeholder for "Factotum inside"](https://github.com/snowplow/sql-runner/issues/61) for SQL Runner Pre the above tickets, some command-line tools will provide a...
This isn't a very popular feature when we have mocked it up, +1 this ticket if you want it...
Currently a step can only depend on a step which the parser has previously seen. For example; ``` step a step b step c