Enforce deprecation message format with EOL for google provider package

[moiseenkov]

This PR introduces unification for deprecation messages formatting within Google's provider package by following:

1. Updated the pre-commit hook check-code-deprecations so it enforces a specific message structure containing the end of life date for the deprecated entity. According to the deprecation policy discussed here, this deadline doesn't enforce removal of the deprecated entity from the source code, it only increases transparency and predictability for users, so they knew how much time do they have for the changes on their side.
2. Another reason for this change is preparation for the cleaning-up process as Google's provider package has collected a lot of old legacy code and we'd be happy to finally clean it up.
3. These changes work only within an airflow/providers/google scope.
4. The default EOL is at least six months ahead.
5. Some classes and methods have assigned earlier deadlines because they have been deprecated for a very long time already and can be removed earlier or their API is shut down and they are not working already.

[potiuk]

cc: @dstandish -> You wanted to do something like that - I think the approach proposed by the Google team (for now in google provider is really nice and we could apply it everywhere)

[ashb]

We should use yyyy-mm-dd (or yyyy.mm.dd, don't mind separator) so the date is unabigious

[ashb]

Instead of a pre-commit check and enforcing the style of the message did you consider adding extra (required) args to the deprecated decorator:

    @deprecated(
        reason="Please use `create_build_without_waiting_for_result`",
        removed_after="2025-03-01",
        category=AirflowProviderDeprecationWarning,
    )

That way the message can be generated programaticaly and be consistent that way -- no need for a precommit check then?

[kacpermuda]

Any standarization to deprecations is a good idea 🚀

did you consider adding extra (required) args to the deprecated decorator

Something related and still on my todo list (for whenever i have time) is #36952 where i wanted to add a custom decorators (or other construct) for different deprecations (function, arg, arg_value etc.), where we could separate these specific parts (what is deprecated, since when, removal date, what to use instead etc.) into separate arguments so it's easier to generate docs out of it. I think it could work even better now with common.compat provider, maybe we could implement something more global for the airflow codebase 😄

[potiuk]

Instead of a pre-commit check and enforcing the style of the message did you consider adding extra (required) args to the deprecated decorator

Pre-commit check is still needed regardless - because we want to make sure that all deprecations have it. But yes - they could be better structured. one problem with adding new fields - is that since providers and airflow core are separated, anything "common" we come up with will have to go to "common.compat" - especially in the light that providers should be compatible with both Airflow 2 and Airflow 3.

So generally - yes better structure at the expense of more cross-provider dependencies.

[kacpermuda]

is that since providers and airflow core are separated, anything "common" we come up with will have to go to "common.compat" - especially in the light that providers should be compatible with both Airflow 2 and Airflow 3.

It's also explained in this discussion: https://github.com/apache/airflow/pull/37075#discussion_r1484366520, maybe someone will find it useful, so just leaving it here.

[potiuk]

It's also explained in this discussion: #37075 (comment), maybe someone will find it useful, so just leaving it here.

Yep. We've been discussing this :)

[moiseenkov]

Instead of a pre-commit check and enforcing the style of the message did you consider adding extra (required) args to the deprecated decorator:

    @deprecated(
        reason="Please use `create_build_without_waiting_for_result`",
        removed_after="2025-03-01",
        category=AirflowProviderDeprecationWarning,
    )

That way the message can be generated programaticaly and be consistent that way -- no need for a precommit check then?

This is the perfect approach, and I'd be happy to apply it, but as it was discussed in https://github.com/apache/airflow/pull/37075#discussion_r1484366520, implementing this new decorator on the Airflow core side would make it very difficult to use in providers as they would have to bump Airflow version in their requirements.

The idea of releasing such a decorator in a separated provider package seems like a great solution! Is there any chance to implement it any time soon? I see that this work has already started #36952. @kacpermuda , @potiuk

Otherwise, our team would be happy to merge the current PR (with date format fixed) now in order to bring some order to the google provider package now. And once the new provider with the structured deprecation decorator is released, we could adopt the current changes accordingly.

[potiuk]

The idea of releasing such a decorator in a separated provider package seems like a great solution! Is there any chance to implement it any time soon? I see that this work has already started https://github.com/apache/airflow/pull/36952. @kacpermuda , @potiuk

The common.compat provider is already there. It's really a matter of:

• adding the decorator there (with a note which minimum versio of compat provider has it).
• bumping minor version of the compat provider
• add a dependency in google provider to common.compat>=NEW_VERSION
• any other provider that will want to use it, will have to do the same

We could also add this decorator to "airflow" utils with the note that it will be usable when min airflow in providers will be >= 3.0 (but then it should be added in airlfow-sdk (cc: @ashb) - so we should not do it now, but possibly leave a TODO to add it there.

[kacpermuda]

The common.compat provider is already there.

My main concern, though I’m not sure if it’s entirely valid, is that we previously discussed having a common.utils provider specifically designed to make it easier to add simple functionality, without external dependencies, to all providers, so that they do not have to rely on core Airflow version. The common.compat provider, as I understand, is meant to serve as a “proxy” that ensures code within providers functions without errors, with potentially fewer features, regardless of the Airflow version or other provider versions, but we should gradually remove things from there as we raise min. dependency versions f.e. for Airflow core. So it looks similar, but I'm not quite sure it's the same. Introducing this type of deprecation module into common.compat could make it more multi-functional, and it can be beneficial, I'm just not sure if i understand the background logic behind common.compat

[potiuk]

So it looks similar, but I'm not quite sure it's the same. Introducing this type of deprecation module into common.compat could make it more multi-functional, and it can be beneficial, I'm just not sure if i understand the background logic behind common.compat

It is the same. I imagine (see above) that the decorator WILL become part of "airflow's sdk" eventually (whether it will be named as such or "task.sdk" - generally in Airflow 3 this SDK will become the "util" equivalent - and this SDK will contain everything that provider should be able to use (and in Airflow 3 providers will have no dependency on Airflow "core" - they will have dependency on the "sdk". So in this case "compat" is precisely compatibilty shim for the future "common" SDK that airlfow providers will be able to use.

The only problem is that we do not YET have the SDK - this is being worked on as part of AIP-72 and will likely come together with restructuring of whole Airlfow repo - including moving providers to separate sub-projects and so on, so it makes very little sense to implement it now in airflow 3, because it will anyhow change.

@ashb - do you agree with that assessment ?

And note for the future - since you proposed it here - we have to be aware that any kind of the common features that are supposed to be used in providers will have to eventually land in "airflow.sdk" that should be always future-compatible and for now they should land also in "common.compat" if we want to implement it in current providers, since we want the providers to work for both Airlfow 2/3.

Does it make sense what I described here? @ashb - is this your understanding as well?

[moiseenkov]

@kacpermuda , hi, Considering the conversation here it seems that the best way for us is promoting your implementation https://github.com/apache/airflow/pull/36952 (as it is almost complete) and release it in the common.compat provider package. Would you have time for re-opening your PR and adding support for sunset dates?

[kacpermuda]

@moiseenkov, I’m happy to revisit #36952, but unfortunately, I can’t commit to a specific timeline at the moment. If you have any available resources, feel free to pick it up in the meantime. Regardless, great job with this PR and moving us closer to more standardized deprecations! :)

[potiuk]

@moiseenkov, I’m happy to revisit #36952, but unfortunately, I can’t commit to a specific timeline at the moment. If you have any available resources, feel free to pick it up in the meantime. Regardless, great job with this PR and moving us closer to more standardized deprecations! :)

I think we can also make it in stages.

Stage 1. Implement it in google provider (with google-specific deprecated decorator - with time and pre-commit checks. Stage 2. Learn from it and move it (possibly extending this to other cases like version mentioned by @dstandish ) -> either to common.compat or future airflow-sdk if it's ready and depended on by providers by that time.

Google provider is big enough to start "some" standardization efforts - many cases and likely > 50% of all operators and there is a history of things started there and adopted by others and made "common" (system tests for example)

I don't think we should aim to have a final solution implemented from day 0 that is applicable to all providers - having google as a good start and "experiment" is IMHO - totally acceptable.

WDYT others?

[moiseenkov]

Hi everyone, As we agreed, I introduced a custom @deprecated decorator within the Google provider package with a detailed structure and pre-commit hook that enforces a sunset date for changes within the provider. The date format is also adjusted.

[VladaZakharova]

Hi @potiuk @eladkal @ashb ! I think we can try to merge this,m WDYT? The longer we wait, the more information about those deprecations become irrelevant haha

[potiuk]

I am all for it @eladkal @dstandish - I think you had doubts - but I think we can do it as 'google provider specific' for now. I guess small, provider-specific solutions like that are better than inaction and waiting for responses. We can always change it later - no harm done to any other providers.

Will merge if I don't hear a complain