Pydantic v2 and Airflow 2.8

[nickolasrm]

Affected module Does it impact the UI, backend or Ingestion Framework?

Ingestion framework

Describe the bug A clear and concise description of what the bug is.

Since the launch of Airflow 2.8 they dropped support for Pydantic v1. Because of that, openmetadata ingestion cannot be installed alongside Airflow which is a common scenario considering the installation of the lineage backend.

To Reproduce

Screenshots or steps to reproduce

Airflow 2.8 release notes

Openmetadata setup.py

Expected behavior A clear and concise description of what you expected to happen.

Make openmetadata ingestion to accept both v1 and v2 versions. Pydantic offers backwards compatibility by using the pydantic.v1 module, so it will be a matter of using the correct syntax for v1 and v2 through the use of if elses on the __version__ variable.

For most models there will be no difference, but for the custom type in the pydantic openmetadata custom types module there may be a big change.

Version:

• OS: [e.g. iOS] Ubuntu 22.04
• Python version: 3.11.7
• OpenMetadata version: [e.g. 0.8] 1.2.5
• OpenMetadata Ingestion package version: [e.g. openmetadata-ingestion[docker]==XYZ] 1.2.5

Additional context Add any other context about the problem here.

I know airflow 2.8 is still incompatible with openmetadata ingestion, but it would be important to migrate or to make it compatible with both v1 and v2 in order to follow project and lib changes. Airflow is just one problem, but projects that use openmetadata may get incompatibility issues due to this outdated dep.

An example of compatiblity with v1 and v2 at the same time is bentoml.

[colax94]

Hello,might be unrelated but I am also seeing issues with Airflow 2.8.1, where OpenMetadata auto generated DAGs are not being scheduled by Airflow. The CRON schedule appears correctly in the UI, but the "Next run date" remains None, and the DAG is never scheduled. The DAGs can still be run manually though.

[sunchao]

Also looking forward for this. We have Airflow 2.8.x internally and would like to use OpenMetadata with it.

[harshach]

hi @sunchao , we will be prioritizing this for 1.5.0 release. Let us know if you can help us providing with any PRs

[sureshms]

@sunchao, how are you? 🤝

[sunchao]

Hi @sureshms @harshach how are you doing? long time no see!

Internally we're just getting started with OpenMetadata. We already have a Airflow 2.8.x setup running and because of the 2.6.x limitation, we had to run a separate Airflow service just for OpenMetadata which is not very desirable, and hence the ask.

Do you guys have a rough timeline for 1.5.0? Also it'd be great if you have some umbrella ticket to track the tasks for unblocking this upgrade. I can't promise I'll have enough time to work on those, but I'll give it a try if I do. 😂

[DataOps7]

Hey @harshach ! We are also blocked on this issue :( You guys can also upgrade pydantic to v2 but still use from pydantic.v1 import ... while you're migrating. Thanks!