apache
PoC of AIP-69 - A first task can be called remote
This PR is a DRAFT to further discuss technical details on AIP-69 direction. DO NOT MERGE - but please feedback on it.
Please treat this as a PoC, it is only partly working. It can be used to distribute the first task but a lot of core required functions to make a MVP are missing. Status of implementation is tracked in airflow/providers/remote/TODO.md in this PR.
If you want to test it you can:
- Checkout the branch
- Run
breeze start-airflow --python 3.12 --load-example-dags --backend postgres --executor RemoteExecutor --answer y - Start the Remote executor with
airflow/providers/remote/_start_remote_worker.sh - A (non templated, not mapped, non failing) task can be run, e.g. use example DAG
example_python_operator. Wohoo! - If you open the UI you will see an additional plugin loaded in the menu "Admin" -> "Remote Worker Status":
Based on progress and time I will sometimes rebase and push changes to this branch. But it will not be something that will be review-able. I will not add any migration logic in case of updates for the additional tables in DB Scheme. If I update the draft PR and DB schema changes, DB might need to be reset. Of course migration support would need to be added before a "real" release.
