airflow icon indicating copy to clipboard operation
airflow copied to clipboard

Have SLA docs reflect reality

Open notatallshaw-gts opened this issue 3 years ago • 3 comments

What do you see as an issue?

The SLA documentation currently states the following:

An SLA, or a Service Level Agreement, is an expectation for the maximum time a Task should take. If a task takes longer than this to run...

However this is not how SLAs currently work in Airflow, the SLA time is calculated from the start of the DAG not from the start of the task.

For example if you have a DAG like this the SLA will always trigger after the DAG has started for 5 minutes even though the task never takes 5 minutes to run:

import datetime

from airflow import DAG
from airflow.sensors.time_sensor import TimeSensor
from airflow.operators.python import PythonOperator


with DAG(dag_id="my_dag", schedule_interval="0 0 * * *") as dag:
    wait_time_mins = TimeSensor(target_time=datetime.time(minute=10))
    run_fast = PythonOperator(
        python_callable=lambda *a, **kw: True,
        sla=datetime.timedelta(minutes=5),
    )
    run_fast.set_upstream(wait_time_mins)

Solving the problem

Update the docs to explain how SLAs work in reality.

Anything else

No response

Are you willing to submit PR?

  • [X] Yes I am willing to submit a PR!

Code of Conduct

notatallshaw-gts avatar Sep 21 '22 16:09 notatallshaw-gts

Feel free to update the docs.

potiuk avatar Sep 21 '22 16:09 potiuk

Feel free to update the docs.

Will do, made issue mostly to see if someone would disagree with me. I'll make a PR to update the docs soon.

notatallshaw-gts avatar Sep 21 '22 17:09 notatallshaw-gts

assigned you :)

potiuk avatar Sep 22 '22 11:09 potiuk