could we use DagFactory in AWS MWAA?.

[muthuramanm]

1.could we use DagFactory in AWS MWAA?. 2.could you please provide athena operator for quering?.. Please Reply.

[SheldonGrant]

@muthuramanm you can write a custom operator based on boto3 and the pythonOperator. We have something like this in our eco-system. http://airflow.apache.org/docs/apache-airflow/stable/howto/custom-operator.html

[SheldonGrant]

@muthuramanm Idea around using dagfactory in AWS MWAA would be amazing. However, the only way to begin to support this in my research is to resolve this issue 68. Since a managed airflow like MWAA or even cloud composer would not allow the current approach to work. Plus, there is the dependency issue.

[muthuramanm]

Yes, I'm using DagFactory in AWS MWAA. But it seems difficult to customize it further.

[lsaiken]

@muthuramanm Which version of AWS MWAA are you using? I'm using Airflow v2.0.22. With the example DAGs, I get the following errors:

Broken DAG: [/usr/local/airflow/dags/example_dag_factory.py] Traceback (most recent call last):
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/dagfactory/dagbuilder.py", line 10, in <module>
    from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
  File "/usr/local/lib/python3.7/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 23, in <module>
    from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator  # noqa
ModuleNotFoundError: No module named 'airflow.providers.cncf'

Broken DAG: [/usr/local/airflow/dags/example_customize_operator.py] Traceback (most recent call last):
  File "/usr/local/airflow/.local/lib/python3.7/site-packages/dagfactory/dagbuilder.py", line 10, in <module>
    from airflow.contrib.operators.kubernetes_pod_operator import KubernetesPodOperator
  File "/usr/local/lib/python3.7/site-packages/airflow/contrib/operators/kubernetes_pod_operator.py", line 23, in <module>
    from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator  # noqa
ModuleNotFoundError: No module named 'airflow.providers.cncf'

[aramakrishnan19]

Dag Factory works in MWAA, here is a sample requirements.txt for 2.0.2

--constraint "https://raw.githubusercontent.com/apache/airflow/constraints-2.0.2/constraints-3.7.txt"
apache-airflow[cncf.kubernetes]
dag-factory==0.11.1