luigi
luigi copied to clipboard
spotify
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in.
I've been trying to use the Kubernetes Job wrapper, but I am facing a task failure, even though the Job executes just fine after it has been spun up. ```...
This uses PEP593 Annotations to support typing of parameters in a way that will make mypy & co treat the instance variables correctly.
This fixes #2879 for me. I used the method described by @hirolau (and the slight correction made by @DVlahovic) This (and one other change, PR #3183) were required for luigi=3.1.0...
In the following code all ChildTasks priority is set to their tId, but it actually only works for the ChildTasks between ParentTask with TaskNr 0 and 1. All ChildTasks between...
The `luigi.contrib.azureblob` depends on the old version of the `azure.storage.blob` package. It needs to be rewritten/updated to support the 12.x.y-series of `azure.storage.blob`. @manish, @ckiosidis : You are listed as contributers....
This PR is meant as a continuation of #3178 and further improves the control over dynamic task requirements handled by the worker. ## Description I added a shallow class `DynamicRequirements`...
## Description `autoload_range` set in `luigi.cfg` emits `UnconsumedParameterWarning` as value is used only in `luigi.__init__`. I've added internal field `_ignore_unconsumed` to ignore this parameter in `core` class. ## Motivation and...
## Description I've changed `luigi.task.flatten_output` implementation to deduplicate outputs from tasks which are required by more than one wrapper task. Implementation is slightly changed (from DFS to BFS), but there's...