luigi
luigi copied to clipboard
[Contrib] Task trigger solution
We use Luigi to run one-off data processing pipelines, and need a way to deploy and trigger these. As luigi doesn't include this, I've coded up a solution that lets you deploy code in virtualenv and trigger luigi tasks inside those virtualenvs using yaml files.
https://github.com/emerald-geomodelling/poltergust
Would there be an interest in including something like this in luigi.contrib? Are there other similar attempts I should be aware of?