airflow-declarative
airflow-declarative copied to clipboard
rambler-digital-solutions
Airflow declarative DAGs via YAML
This PR adds support for airflow 2, and: - Drops support for airflow 1 (to avoid complicating the tests, because `airflow.operators.dummy` is not importable in v1, and `airflow.operators.dummy_operator` is not...
Trafaret checks schema for operator and sensors in do template. It limits templating. This PR adds support for invalid schema at templating, but after transform it ensures schema.
Airflow declarative supports check callback for importable. If i have different environments for airflow workers i can't build my dags. I want the solution which can work with checking importable...
yaml.dump at the end of stream always add '\n...\n' it can't be disabled and it cause errors when templating something in strings. This PR fixes this behavior.
We need document what we have here. Currently all we have is reference to source code. That's geeky, but not user friendly. Documentation should cover at least the following topics:...
This PR contains a couple of commits containing failing tests for the two bugs in Declarative I'm aware of: 1. The `do` block strips newlines from the substituted jinja variables,...
Since we operate with plain data, we can easily detect duplicate tasks across multiple DAGs. For instance, several of our DAGs may have the same sensor which awaits the same...
Case: we have a scheduler service which runs under some specific Python version. Say 2.7. Now we need to run DAGs for Python 3.x projects. Since scheduler does class/callback import...
For now it's possible to specify function or callable-class as a callback. For some cases it's handly to set classmethod or staticmethod as a operator callback. It's actually more important...
Metadata
Owner
Metadata
Airflow declarative DAGs via YAML