[KED-1541] DockerOperator workflow

[seeM]

Description

Use DockerOperators that run commands of the project's docker image, instead of PythonOperators.

Context

It greatly simplifies dependency management: dependency management is "outsourced" to DAG creation rather than DAG deployment – the deployment mechanism doesn't have to juggle virtual environments and PYTHONPATHs. This is particularly important when managing DAGs across many projects with potentially conflicting requirements.

Possible Implementation

I haven't looked extensively, but I imagine it could be implemented by subclassing AirflowRunner or AbstractRunner with a DockerAirflowRunner with its own dag_template. I imagine the logic should be quite straightforward. But it might require refactoring and renaming existing modules and classes to allow differentAirflowRunners in a sensible way.

We'd also need to be careful around making this work seamlessly with kedro-docker.

Possible Alternatives

I might resort to manually building out my DAGs using the DockerOperator approach, which is what I'm doing without kedro already. But that's not a great alternative.

This could be handled in another plugin, like kedro-docker-airflow, but I think there'd be a fair amount of code re-use with kedro-airflow, so maybe not a great idea.

[seeM]

@921kiyo any thoughts on this?

[921kiyo]

@seeM Apologies for the late response. We discussed this internally, and are planning to rearchitect the Kedro-Airflow architecture, and we will think about how to make users configurable the Airflow operator types in their sides.

Unfortunately, however, it's not in our priority at the moment. And we will definitely come back to this later :) As a temporary workaround until we change it, you could manually build your DAGs (as you are doing now), or inherit AirflowRunner and override _run method.

Thank you for the feedback! We are more than happy to hear other use cases that the plugins might not fit (if any), so we can consider that as well :)