apache
feat: Add custom provider runtime checks
As discussed here at pr #39513, changes introduced by #39336 make all previous OpenLineage provider versions incompatible with Airflow >= 2.10.0 (due to changes in try_number behaviour change in core Airflow).
Using previous version of provider is still possible for Airflow >= 2.10.0, but will cause invalid behaviour (yet it will not cause any errors). We can't modify already released versions and provide a max supported Airflow version so we'd like to have a runtime check to ensure that a compatible version of OL provider is installed
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.
Also - we need to actually bump openlineage to 1.8.0 in provider.yaml to make the Provider tests pass (it failed now correctly detecting wrong version :D).
I was wondering how to do it aside the release process not to break anything, as doing it by hand seemed really wrong to me . I modified the changelog, init and provider.yaml, is that the correct approach ? I want to make sure that for the next release it will not get bumped again, and in result the 1.8.0 will not exist.
BTW. Both workflows aare fine. It's generally OK to bump minor version of package if we know change is breakin. The breeze release-management prepare-provider-documentation will see it and rather than bumping it, it will just tell release manager "hey this package is going to be relased in new major version` .
Also, where are we with Airflow 2.10 on timeline? Do you think there will be time for next providers release before the 2.10. release? We need to make sure that, by accident, we don't get the apache-airflow-providers-openlineage < 1.8.0 in Airflow 2.10 constraints 😄
Also, where are we with Airflow 2.10 on timeline? Do you think there will be time for next providers release before the 2.10. release? We need to make sure that, by accident, we don't get the apache-airflow-providers-openlineage < 1.8.0 in Airflow 2.10 constraints 😄
No worries, that will take quite some time (months rather than weeks).
@potiuk @eladkal It's all green, so if there are no more things to do here i think it can be merged.
I think we should also update apache-airflow-providers-openlineage==1.8.0 the constraint in constraint-main for all Python versions.
I think we should also update apache-airflow-providers-openlineage==1.8.0 the constraint in constraint-main for all Python versions.
constraints-main is updated automatically. sometimes it takes time for the job to be completed
constraints-main is updated automatically. sometimes it takes time for the job to be completed
Seems that they have not been updated for a while, there were few main failures to prevent it but likely after https://github.com/apache/airflow/pull/39747 we are likely to get them updated