Hybrid Executor Aliasing raises exception

[HadarSha]

Apache Airflow version

2.10.0

Airflow Helm Chart version

1.15.0

What happened?

When trying to use hybrid executors and aliasing one of them (in my case AWS ECS Executor), I get an error when Airflow is being initialized (my deployment uses the Airflow Official Helm Chart on AWS EKS, this exception happens in the airflow-run-migrations pod).

My offending configuration bit is the following environment variable trying to set the AwsEcsExecutor to have an alias of FargateExecutor :

env:
  - name: "AIRFLOW__CORE__EXECUTOR"
    value: "CeleryExecutor,KubernetesExecutor,airflow.providers.amazon.aws.executors.ecs.AwsEcsExecutor:FargateExecutor"

The traceback I am getting is as follows:

....................

Last check result:
$ airflow db check
/home/airflow/.local/lib/python3.12/site-packages/airflow/plugins_manager.py:30 DeprecationWarning: 'cgitb' is deprecated and slated for removal in Python 3.13
Traceback (most recent call last):
  File "/home/airflow/.local/bin/airflow", line 5, in <module>
    from airflow.__main__ import main
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/__main__.py", line 38, in <module>
    from airflow.cli import cli_parser
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/cli/cli_parser.py", line 62, in <module>
    for executor_name in ExecutorLoader.get_executor_names():
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/executors/executor_loader.py", line 150, in get_executor_names
    return cls._get_executor_names()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/airflow/.local/lib/python3.12/site-packages/airflow/executors/executor_loader.py", line 111, in _get_executor_names
    raise AirflowConfigException(
airflow.exceptions.AirflowConfigException: Incorrectly formatted executor configuration. Second portion of an executor configuration must be a module path or plugin but received: FargateExecutor
2024-08-27T14:53:00.179007881Z

What you think should happen instead?

Based on the official Airflow documentation for using Hybrid executors with aliasing, my configuration should be valid and I expect the ECSExecutor to be available for use as FargateExecutor as per my alias given in the configuration. See relevant part of the documentation with example:

To make it easier to specify executors on tasks and DAGs, executor configuration now supports aliases. You may then use this alias to refer to the executor in your DAGs (see below).

[core]
executor = 'LocalExecutor,my.custom.module.ExecutorClass:ShortName'

How to reproduce

1. Make sure you are using Airflow Helm Chart 1.15.0 and Airflow Version 2.10.0
2. Make sure Celery and Kubernetes providers are also installed, as I was using the CeleryKubernetesExecutor as my default executor (though this might not be critical for testing the alias issue per se) - e.g. pip install apache-airflow-providers-celery==3.8.0 apache-airflow-providers-cncf-kubernetes==8.4.0)
3. Install the Amazon Provider package version 8.28.0 (e.g. pip install apache-airflow-providers-amazon==8.28.0)
4. Set the executor key in the Helm chart to be CeleryKubernetesExecutor
5. Set the following environment variable: AIRFLOW__CORE__EXECUTOR=CeleryExecutor,KubernetesExecutor,airflow.providers.amazon.aws.executors.ecs.AwsEcsExecutor:FargateExecutor
6. Make sure the migrateDatabaseJob is enabled in the Helm chart
7. Deploy Airflow and watch the migrations job when it runs as it's where I had encountered my issue (but it might pop up in other places as well)

Operating System

Debian GNU/Linux 12 (bookworm)

Versions of Apache Airflow Providers

apache-airflow-providers-amazon==8.28.0 apache-airflow-providers-atlassian-jira==2.7.0 apache-airflow-providers-celery==3.8.0 apache-airflow-providers-cncf-kubernetes==8.4.0 apache-airflow-providers-common-compat==1.1.0 apache-airflow-providers-common-io==1.4.0 apache-airflow-providers-common-sql==1.15.0 apache-airflow-providers-fab==1.2.2 apache-airflow-providers-ftp==3.10.1 apache-airflow-providers-google==10.19.0 apache-airflow-providers-http==4.12.0 apache-airflow-providers-imap==3.6.1 apache-airflow-providers-jenkins==3.7.0 apache-airflow-providers-openlineage==1.10.0 apache-airflow-providers-postgres==5.12.0 apache-airflow-providers-salesforce==5.8.0 apache-airflow-providers-slack==8.9.0 apache-airflow-providers-smtp==1.7.1 apache-airflow-providers-sqlite==3.8.2 apache-airflow-providers-tableau==4.6.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

K8s deployment on AWS EKS, offending pod is airflow-run-migrations (AKA migrateDatabaseJob in the Helm chart), the executor being aliased in my case is the AwsEcsExecutor from the apache-airflow-providers-amazon package.

Airflow Helm Chart Version - 1.15.0

Anything else?

No response

Are you willing to submit PR?

• [ ] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[boring-cyborg[bot]]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[jscheffl]

Hybrid execution is (currently) not supported in the Airflow Helm chart. That is still under development.

[jscheffl]

See https://github.com/apache/airflow/pull/41524