Getting error while deploying the ingestion dag file for Airflow

[koabhi]

Affected module Does it impact the UI, backend or Ingestion Framework? Ingestion Framework

Describe the bug am trying to ingest the metadata from aws glue using airflow sdk am referring to: https://docs.open-metadata.org/openmetadata/connectors/database/glue/airflow for that when am running the ingestion_dag.py script with required yaml configuration, am getting the error of: airflow.exceptions.AirflowException: Invalid values of dag.default_view: only support ['tree', 'graph', 'duration', 'gantt', 'landing_times'], but get grid

To Reproduce

Screenshots or steps to reproduce

Expected behavior A clear and concise description of what you expected to happen.

Version:

• OS: [e.g. iOS] : 12.5
• Python version: 3.9
• OpenMetadata version: 0.11.5.3
• OpenMetadata Ingestion package version: 0.11.5.3

Additional context Add any other context about the problem here.

[pmbrull]

hi @koabhi could you please give us some more details on your setup? Are you using the ingestion docker image we provide or are you running with your own Airflow? If so, which Airflow version are you using?

Thanks!

[koabhi]

hello @pmbrull am using kubernetes to deploy open-metadata on AWS EKS Cluster. Aws EKS Cluster, Elasticsearch and RDS are configured, up and running. But when am trying to automate the metadata ingestion using airflow, am following the documentation: https://docs.open-metadata.org/openmetadata/connectors/database/glue/airflow but while deploying the dag file am getting this error: airflow.exceptions.AirflowException: Invalid values of dag.default_view: only support ['tree', 'graph', 'duration', 'gantt', 'landing_times'], but get grid

When am hosting the open-metadata on localhost and trying to deploy a pipeline for my aws glue catalog, am getting the following issue on localhost server:

[pmbrull]

Hi @koabhi, on EKS did you deploy our Ingestion Container or are you configuring your own Airflow instance?

The error shown here says that Airflow cannot be reached based on your OM configurations.

For k8s, we have these docs which should deploy ingestion (Airflow) pods that should be reachable from the OM server.

If instead you are using your own Airflow instance, which version of Airflow are you using?

[koabhi]

Hello @pmbrull , I'm following the docs to configure airflow for the Ingestion Container. If you want I can share with you my open-metadata-dependencies.yaml file, openmetadata.yaml and permissions.yaml file with you.

[pmbrull]

Hi @koabhi thanks for the info. If possible we could continue this conversation in Slack in the #support channel and go over the details. It might be faster!

Thanks!

[pmbrull]

From support https://openmetadata.slack.com/archives/C02B6955S4S/p1662892638627169

[pmbrull]

connected with @koabhi and @akash-jain-10. Looks like a configuration issue, we'll be waiting for more information

[harshach]

@koabhi can you confirm if you are able to resolve this issue

[harshach]

@koabhi Please let us know if you are able to get past the issues you are seeing

[pmbrull]

closing for inactivity. @koabhi feel free to reopen if needed