openlineage: migrate OpenLineage provider to V2 facets.

[JDarDagran]

In https://github.com/OpenLineage/OpenLineage/pull/2520 released in openlineage-python==1.13.1 there were introduced V2 classes for facets and events.

[JDarDagran]

I ran breeze testing tests --upgrade-boto tests/providers/docker in main branch with latest CI image - getting same error results. Celery integration test seems flaky, no?

[potiuk]

I ran breeze testing tests --upgrade-boto tests/providers/docker in main branch with latest CI image - getting same error results.

https://github.com/apache/airflow/pull/39747 should fix the docker test in "Latest Botocore".

[mobuchowski]

@JDarDagran we probably need to bump cross-provider version dependencies. If someone has older OL provider version, and bumps some of the other provider versions, the openlineage methods will fail.

[potiuk]

As discussed with @mobuchowski -> we need to add back-compatibility try/import to past versions of the openlineage provider. We could even potentially add back-compatibility test suite for older version of of openlineage client < 1.13 to test it (symiliar to what we have in Pydantic/Botocore cases.

[JDarDagran]

I've added imports with weird-looking structure at first glance:

if TYPE_CHECKING:
    # v2 imports
else:
   try:
       # v2 imports
   except ImportError:
       # v1 imports

This satisfies mypy and works with previous OL provider versions. I'll be adding backwards-compatibility cross-provider checks in separate PR soon.

[potiuk]

Should we re-start discussion about common provider for that one? I have an idea..

[potiuk]

Discussion at devlist https://lists.apache.org/thread/hjmrv1wc2rxhkvozpvccs4mhgwg5sw2f

[potiuk]

My proposal is to wait for LAZY CONSENSUS on compat provider and implement it (See https://lists.apache.org/thread/hjmrv1wc2rxhkvozpvccs4mhgwg5sw2f)

[JDarDagran]

yep, waiting for better times :)

[potiuk]

Consensus thread started ...

[potiuk]

PR to add common.compat here https://github.com/apache/airflow/pull/40374 @kacpermuda @JDarDagran

[dolfinus]

Hm, openlineage-python client minimal version is currently 1.16.0: https://github.com/apache/airflow/blob/ed57711866527d4e56ce195944b26607317b1f26/airflow/providers/openlineage/provider.yaml#L49

Why need adding these try: ... except: for handling versions <1.13.1?

[potiuk]

Hm, openlineage-python client minimal version is currently 1.16.0:

https://github.com/apache/airflow/blob/ed57711866527d4e56ce195944b26607317b1f26/airflow/providers/openlineage/provider.yaml#L49

Why need adding these try: ... except: for handling versions <1.13.1?

Because providers (for example AWS) might be installed on older airflow versions AND with older openlineage provider version.

[potiuk]

You have to remember that what you see now in main is the LATEST set of all providers and latest airflow. This is why we have compatibility and lowest-direct dependencies tests to test current latest code assuming that older versions of airlfow and older versions of providers might be installed.

[dolfinus]

to test current latest code assuming that older versions of airlfow and older versions of providers might be installed

How is this related to adding a bunch of try..except for missing imports? I don't get it.

[potiuk]

pip install apache-airflow==2.7.0 apache-airflow-providers-openlineage==1.5.0 apache-airflow-providers-amazon==NEXT_VERSION

And then try without try ... except

[dolfinus]

Ok, got it, thanks for clarifying!

[potiuk]

And https://github.com/apache/airflow/pull/40374 merged. You can rebase and move the code there (and add dependency on apache-airflow-providers-common-compat

[eladkal]

@JDarDagran can you rebase?

[JDarDagran]

@eladkal I'll be working on moving part of the code to common.compat provider

[potiuk]

@eladkal I'll be working on moving part of the code to common.compat provider

Cool :)

[potiuk]

almost :)

[JDarDagran]

I think I need to drop tests checking what is imported with different availability of OL client. mocking imports breaks other tests when running in parallel.

[potiuk]

I think I need to drop tests checking what is imported with different availability of OL client. mocking imports breaks other tests when running in parallel.

The OTEL "lowest dependencies" test is - quite surprisingly caused by packaging version - I have a PR for that https://github.com/apache/airflow/pull/40865 - but it might have some side-effects so I am waiting for it to complete.

[potiuk]

I found it by bisecting downgraded packages BTW.

[JDarDagran]

I think I need to drop tests checking what is imported with different availability of OL client. mocking imports breaks other tests when running in parallel.

The OTEL "lowest dependencies" test is - quite surprisingly caused by packaging version - I have a PR for that #40865 - but it might have some side-effects so I am waiting for it to complete.

That's great to have it fixed. Still, I meant different way of tests breaking caused by mocking builtin import - which I removed eventually. Now you can see it's almost green.

[potiuk]

Let's see -rebased to take latest deps into account :crossed_fingers:

[JDarDagran]

Last one seems flaky?

[JDarDagran]

@potiuk looks like I'm on green path finally consistently 🟢 Would love to get this merged soon as new features arrive and I still need to rebase and apply new imports :)