Run tests for Providers also for Airflow 2.8

[potiuk]

This is a follow-up on https://github.com/apache/airflow/pull/39513 to add support for running Provider tests against Airlfow 2.8 installed from PyPI.

---

^ Add meaningful description above Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

OK. Just 4 tests left to fix in 2.8 :).

[potiuk]

The change for tests for Providers with Airflow 2.8 compatibility should be ready. That was fairly easy - there is only one case where FileTransfer operator looks like to be broken when openlineage is enabled. I disabled it for 2.8, but @mobuchowski @kacpermuda - maybe you would be able to fix it in the provider?

Here is the error when I re-enable the test:

_________________________________________________________________________________ test_get_openlineage_facets_on_start _________________________________________________________________________________

    def test_get_openlineage_facets_on_start():
        src_bucket = "src-bucket"
        src_key = "src-key"
        dst_bucket = "dst-bucket"
        dst_key = "dst-key"

        expected_input = Dataset(namespace=f"s3://{src_bucket}", name=src_key)
        expected_output = Dataset(namespace=f"s3://{dst_bucket}", name=dst_key)

        op = FileTransferOperator(
            task_id="test",
            src=f"s3://{src_bucket}/{src_key}",
            dst=f"s3://{dst_bucket}/{dst_key}",
        )

>       lineage = op.get_openlineage_facets_on_start()

tests/providers/common/io/operators/test_file_transfer.py:69:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/local/lib/python3.8/site-packages/airflow/providers/common/io/operators/file_transfer.py:85: in get_openlineage_facets_on_start
    input_dataset = Dataset(namespace=src.namespace, name=src.key)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = ObjectStoragePath('s3://src-bucket/src-key'), item = 'namespace'

    def __getattr__(self, item: str) -> Any:
        if item == "_accessor":
            # cache the _accessor attribute on first access
            kwargs = self._kwargs.copy()
            self._accessor = _accessor = self._default_accessor(self._url, **kwargs)
            return _accessor
        else:
>           raise AttributeError(item)
E           AttributeError: namespace

/usr/local/lib/python3.8/site-packages/upath/core.py:267: AttributeError

[potiuk]

Looks green

[kacpermuda]

there is only one case where FileTransfer operator looks like to be broken when openlineage is enabled. I disabled it for 2.8, but @mobuchowski @kacpermuda - maybe you would be able to fix it in the provider?

@potiuk Could you please tell me which command should i run to reproduce? Are we running the providers from current main against different airflow versions or are we using some past versions of the providers? I see that namespace property has been added here, together with OL support, so I'm trying to understand if there might be some mismatch of the provider versions there.

[potiuk]

@potiuk Could you please tell me which command should i run to reproduce? Are we running the providers from current main against different airflow versions or are we using some past versions of the providers? I see that namespace property has been added here, together with OL support, so I'm trying to understand if there might be some mismatch of the provider versions there.

The exact steps to repro are explained here : https://github.com/apache/airflow/blob/main/contributing-docs/testing/unit_tests.rst#compatibility-provider-unit-tests-against-older-airflow-releases (just use 2.8.4 as airflow version).

The way it is done:

1. providers are prepared from current branch - so essentially main (in dist)
2. when entering breeze, airflow is installed from PyPI (2.8.4 in this case). providers are installed from dist prepared packages
3. tests folder is mounted from current branch tests folder, so you can modify them in-place and tests will be updated also in breeze
4. if you woud like to modify some of the providers in attempt to fix or debug it, the providers are installed as packages, so you have to modify them in installed "site-packages" (maybe in the future we should make it easier to allow to also mount providers instead, but this is a bit tricky.

[potiuk]

together with OL support, so I'm trying to understand if there might be some mismatch of the provider versions there.

I think the problem is connected with upath integration. Airflow 2.8.4 uses older upath version that was not compatible with Python 3.12 - cc: @bolkedebruin

[bolkedebruin]

@potiuk Airflow 2.8.4 requires (constraints) universal-path 0.1.4 which is indeed not compatible with python 3.12. However., I am not sure if this is related. The namespace property was added past in 2.9.0, so I guess the provider is only compatible with Airflow 2.9.0+ without adjustments?

[potiuk]

@potiuk Airflow 2.8.4 requires (constraints) universal-path 0.1.4 which is indeed not compatible with python 3.12. However., I am not sure if this is related. The namespace property was added past in 2.9.0, so I guess the provider is only compatible with Airflow 2.9.0+ without adjustments?

Yes. That was my thought as well (and that's how I solved it in the PR) - In this case I check if we are retrieving openlineage facets in Airlfow < 2.9.0 in this operator and fail it explicitly if so: https://github.com/apache/airflow/pull/39606/files#diff-ee30b541cc8033613196b62a0e5d9ff0276113f27d011a9ffbcd355826205f14R87 . Note that FileTransferOperator should work fine when openlineage is disabled even in 2.8.*

Not sure if this is the best approach @mobuchowski @kacpermuda -> maybe we can make it works also in 2.8 ?

[kacpermuda]

@potiuk do you think #39755 will be enough to fix the test you mentioned and make the io provider + OpenLineage work well with Airflow 2.8 ?

[potiuk]

@potiuk do you think #39755 will be enough to fix the test you mentioned and make the io provider + OpenLineage work well with Airflow 2.8 ?

Yes. In the last rebase I 'lost" previous commit - so you could not really properly test it. but I just restored it and it looks like it's working just fine.

[kacpermuda]

I was wondering why i could not do it. I was trying to run breeze with dev providers and Airflow 2.8.4 like you mentioned, but the fab provider required 2.9.0. After i dealt with it, and run the test, it passed without any problems, but now it makes more sense. Anyway, glad that it should be solved now ^^

[potiuk]

I was wondering why i could not do it. I was trying to run breeze with dev providers and Airflow 2.8.4 like you mentioned, but the fab provider required 2.9.0. After i dealt with it, and run the test, it passed without any problems, but now it makes more sense. Anyway, glad that it should be solved now ^^

Yes. I have to upgrade the docs to add information about removing incompatible providers (and I think I can change it to run those tests locally without actually building packages - so that will simplify the process (separate PR)

[potiuk]

OK . That one is ready to be finally reviewed. All problems with compatibility with 2.8 have been solved in the test suite, also thanks to that one we found out the compatbility issue of common.io filetransfer openlineage integration for 2.8 that has been fixed in https://github.com/apache/airflow/pull/39755 -> so thsoe tests are alredy showing it's worth :rocket:

Once this one is merged, I will work on 2.7 compatibility and - in parallell - making it easier to debug and reproduce the tests with previous version of airflow locally.

[potiuk]

I would love to merge that one and work on 2.7 tests :)

[potiuk]

Anyone :) ? I am already working on 2.7 compatibility in #39862