Airflow Standard Provider

[romsharon98]

Extract all time operators and sensors from airflow core and remove them to standard provider Mailing list thread: https://lists.apache.org/thread/2dmlqkcmyomm4q7rrovygs6bw655zx07

---

^ Add meaningful description above Read the Pull Request Guidelines for more information. In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[potiuk]

One more thing - at least for now, "standard" should be added to list of preinstalled providers in hatch_build.py

PRE_INSTALLED_PROVIDERS = [
    "common.compat",
    "common.io",
    "common.sql",
    "fab>=1.0.2",
    "ftp",
    "http",
    "imap",
    "smtp",
    "sqlite",
]

[potiuk]

We might change where it is added in Airflow 3, but for now it should be added here.

[potiuk]

1. we need to deprecate the existed operators in core

Yes, but I think this should be done as separate PR to v2-10-test. I think we do not want to back-port that one to v2-10-test - and we should simply do it there (after we release standard as provider). I am not sure we want to back-port the whole PR here to v2-10-test (providers are not build nor tested there anyway).

2. we are missing all the operators (currently it's only for time operators)

I think those should be separate, smaller PRs ?

3. I am not sure what about the hooks folder?

I think it's the same ^^ - some should be moved but separately.

[gopidesupavan]

Thanks @romsharon98 , will be adding rest of the operators and sensors to this.

[gopidesupavan]

we are missing few things...

1. we need to deprecate the existed operators in core
2. we are missing all the operators and sensors we have in core (currently it's only for time operators)
3. I am not sure what about the hooks folder?

@eladkal me and romsharon discussed , i will be adding rest of the operators and sensors.

[potiuk]

Ah .. so no ... We should not add it to preinstalled just yet - until we release it (sorry @romsharon98 -> see the failure on openapi client).

[eladkal]

Yes, but I think this should be done as separate PR to v2-10-test. I think we do not want to back-port that one to v2-10-test - and we should simply do it there (after we release standard as provider). I am not sure we want to back-port the whole PR here to v2-10-test (providers are not build nor tested there anyway).

Then I propose this PR to remove the files from main that we migrated to the provider. The PR to v2-10-test would be just to replace the classes with deprecation notes after we release the provider.

[potiuk]

Then I propose this PR to remove the files from main that we migrated to the provider.

This is what the PR does.

[eladkal]

Ah! I didn't see changes to airflow/core folder so I assumed it was missed. Not very friendly UX design from GitHub

[gopidesupavan]

Yes, but I think this should be done as separate PR to v2-10-test. I think we do not want to back-port that one to v2-10-test - and we should simply do it there (after we release standard as provider). I am not sure we want to back-port the whole PR here to v2-10-test (providers are not build nor tested there anyway).

Then I propose this PR to remove the files from main that we migrated to the provider. The PR to v2-10-test would be just to replace the classes with deprecation notes after we release the provider.

Please correct me if i understood this, on v2-10-test deprecation updates required for all the changes we are doing on main?

[potiuk]

Please correct me if i understood this, on v2-10-test deprecation updates required for all the changes we are doing on main?

No - only those that are supposed to affect DAG authors.

[potiuk]

BTW. We will have to make the old "standard" providers still work in Airflow 3 I think so we will have to make the old operator imports work (likely with the mechanism mentioned by @ashb - sys.meta_path) - otherwise it will require many dags to change. I think we should apply the same mechanism as we will do for importing BaseOperator etc. in Airflow 3 (and still raise deprecation warning).

@ashb ? Do you agree ?

[potiuk]

Interesting. The errors we see are very useful :).

1. examples are part of the documentation of "main" airflow" - and I am afraid they will have to remain as examples in "airflow" not moved to provider, especially because point 2)...

2. the examples should be embedded and shown in airflow when "examples" are enabled - I think we should not (at least not now) to develop a mechanism to read examples for Airflow from providers.

3. The tests should also mock the moved packages.

<module 'airflow.sensors.time_delta' from '/usr/local/lib/python3.8/site-packages/airflow/sensors/time_delta.py'> does not have the attribute 'sleep'
FAILED tests/providers/standard/time/sensors/test_time_delta.py::TestTimeDeltaSensorAsync::test_wait_sensor[True] - AttributeError: <module 'airflow.sensors.time_delta' from '/u

[romsharon98]

Interesting. The errors we see are very useful :).

1. examples are part of the documentation of "main" airflow" - and I am afraid they will have to remain as examples in "airflow" not moved to provider, especially because point 2)...
2. the examples should be embedded and shown in airflow when "examples" are enabled - I think we should not (at least not now) to develop a mechanism to read examples for Airflow from providers.
3. The tests should also mock the moved packages.

<module 'airflow.sensors.time_delta' from '/usr/local/lib/python3.8/site-packages/airflow/sensors/time_delta.py'> does not have the attribute 'sleep'
FAILED tests/providers/standard/time/sensors/test_time_delta.py::TestTimeDeltaSensorAsync::test_wait_sensor[True] - AttributeError: <module 'airflow.sensors.time_delta' from '/u

revert the example dags to be under main airflow. should I add a link in index.yaml for those example dags?

also not sure but should I change somewhere in the code to tell that this provider is mandatory and will auto installed?

[potiuk]

also not sure but should I change somewhere in the code to tell that this provider is mandatory and will auto installed?

That's the preinstalled_providers - but it will only work after it's been released and it will likely change with Airflow 3 so for now - no need.

[gopidesupavan]

so for time related things its placed under time/ , for other operators and sensors am placing like this .

standard/core/operators/ standard/core/sensors/

is this fine? or any other suggestion.

[eladkal]

Lets wait with merge till we understand the impact of https://github.com/apache/airflow/pull/41727#discussion_r1730836865

[eladkal]

Lets wait with merge till we understand the impact of #41727 (comment)

relevant code was reverted. We are good to go here.

[romsharon98]

I think that after the consensus we can remove the request change @eladkal.

[potiuk]

LGTM in general. Tests are failing and need to be fixed, and I added few comments that should be resolved before merging that one. Also I think we should figure out (but that might be next) how to get example_dags to display them in Airflow 3 from provider (likely change/extend from where they are loaded) - but this should be a separate PR, likely when we have few more examples moved.

[gopidesupavan]

@romsharon98 just checking is the changes required in here https://github.com/apache/airflow/blob/main/docs/apache-airflow/operators-and-hooks-ref.rst?plain=1#L91 ?

[romsharon98]

@romsharon98 just checking is the changes required in here https://github.com/apache/airflow/blob/main/docs/apache-airflow/operators-and-hooks-ref.rst?plain=1#L91 ?

good catch! I will modify it 😄