airflow Adding sensor decorator

Added the @task.sensor decorator to convert a Python function to an instance of the BaseSensorOperator. Example usage of the decorator is:

@task.sensor(poke_interval=60, timeout=3600, mode="poke")
def f() -> PokeReturnValue:
    # implement the condition 
    condition_met = ...
    operator_return_value = ...
    return PokeReturnValue(is_done=condition_met, xcom_value=operator_return_value)

closes: #20323

^ Add meaningful description above

Read the Pull Request Guidelines for more information. In case of fundamental code change, Airflow Improvement Proposal (AIP) is needed. In case of a new dependency, check compliance with the ASF 3rd Party License Policy. In case of backwards incompatible changes please leave a note in UPDATING.md.

Mar 28 '22 07:03 mingshi-wang

LGTM. But we need one other approve to merge it.

Mar 31 '22 17:03 potiuk

The PR most likely needs to run full matrix of tests because it modifies parts of the core of Airflow. However, committers might decide to merge it quickly and take the risk. If they don't merge it quickly - please rebase it to the latest main at your convenience, or amend the last commit of the PR, and push it with --force-with-lease.

Mar 31 '22 17:03 github-actions[bot]

Is it still in time for 2.3?

I'd say yes - but leave it up to @jedcunningham @ephraimbuddy to decide.

Apr 09 '22 13:04 potiuk

@ashb @jedcunningham @ephraimbuddy -> shall we add it to 2.3 ? It's a pretty requested feature and seems ready (but I leave it up to you to decide :).

Apr 13 '22 16:04 potiuk

Fine with me cc: @jedcunningham

Apr 13 '22 21:04 ephraimbuddy

No objections here 👍

Apr 13 '22 23:04 jedcunningham

Someone will need to check how this interacts with task mapping first please

Apr 14 '22 06:04 ashb

I think this one is a bit risky to add for 2.3.0 so let's iterate and merge after we branch off.

Apr 25 '22 18:04 potiuk

Eagerly waiting for this to be merged and released!

Jun 03 '22 16:06 abhilash1in

@potiuk Is it OK to merge this PR?

Jun 04 '22 16:06 mingshi-wang

@mingshi-wang - docs build is failing (that's something that you should fix first) and rebasing the change would also be useful while you do it.

Also we are waiting for @jedcunningham 's re-review - just re-requested it.

Jun 04 '22 16:06 potiuk

@mingshi-wang any updates? Eagerly waiting for this to be merged and released!

Sep 13 '22 17:09 AndreGodinho7

Rebased it to see if the Helm cheart failures were carried from another error.

Sep 19 '22 13:09 potiuk

Rebased it to see if the Helm cheart failures were carried from another error.

Hey @potiuk The Helm local executor still fails. Could you help me understand what could be the cause of the failure? I cannot find any useful message. Thanks!

Sep 20 '22 01:09 mingshi-wang

I think this was accidental failure - simply a problem with lack of space on the machine running it I rebased it and let's see if it happens again. It might also be that we need to optimise some log handling (it looks like logs were filling the disk space. It also could be that (sometimes it happens) that the change causes some recursive behaviour that simply fills up all available disk while running (by producing continuous stream of logs) - in this case that will be a bug to be fixed by you.

Sep 20 '22 07:09 potiuk

(BTW. The useful message about lack of disk space was in a "cancelled" K8S executor (cancelled because of running too long while trying to work with missing space). The local one was cancelled by K8s failing (we have "fast fail" in this case). Simply GitHub UI shows such "cancelled" jobs less prominently than those that are "failing".

Sep 20 '22 07:09 potiuk

I think our Public runners are not big enough to handle "full tests" with k8s :( . So I removed "full tests needed" label and closed/reopened it to run a smaller set of tests

Sep 20 '22 11:09 potiuk

I think our Public runners are not big enough to handle "full tests" with k8s :( . So I removed "full tests needed" label and closed/reopened it to run a smaller set of tests

Thanks for your help!

Sep 20 '22 15:09 mingshi-wang

HI @jedcunningham I have addressed your comments. Can you take a look again? Thanks!

Sep 20 '22 15:09 mingshi-wang

Hi @potiuk will you be ok to merge this PR? I see all tests passed.

Sep 21 '22 15:09 mingshi-wang

We are waiting for @jedcunningham to confirm that he is OK with it - can't merge it without it.

Sep 22 '22 11:09 potiuk

HI @jedcunningham can you please take a look again? Thanks!

Sep 25 '22 18:09 mingshi-wang

We are waiting for @jedcunningham to confirm that he is OK with it - can't merge it without it.

Hi @potiuk, Jed seems unavailable. Is it possible to get another reviewer?

Oct 03 '22 16:10 mingshi-wang

Hi @jedcunningham - do you have a chance to take a look?

Oct 09 '22 17:10 mingshi-wang

Hi @uranusjr @potiuk I think I need some help merging this PR. @jedcunningham is not available to approve this PR. Would you provide some help wrapping up this PR? Thanks!

Oct 13 '22 17:10 mingshi-wang

I dismissed Jed's review - can you please rebase it @mingshi-wang (there is a conflict) - sorry for long delay - I've been on holidays :).

Oct 23 '22 14:10 potiuk

I dismissed Jed's review - can you please rebase it @mingshi-wang (there is a conflict) - sorry for long delay - I've been on holidays :).

I rebased the PR. Could you help merge it?

Oct 25 '22 16:10 mingshi-wang

Just one more check @uranusjr @jedcunningham ?

Oct 26 '22 01:10 potiuk

@mingshi-wang - I'd love to merge it - can you please address the last comments of @uranusjr ?

Oct 31 '22 04:10 potiuk

@mingshi-wang - I'd love to merge it - can you please address the last comments of @uranusjr ?

Sure will do.

Oct 31 '22 07:10 mingshi-wang