bdit_data-sources icon indicating copy to clipboard operation
bdit_data-sources copied to clipboard

812 eoy table

Open chmnata opened this issue 1 year ago • 3 comments

What this pull request accomplishes:

  • taskflowify-ish the dag (definitely more things can be done on the bluetooth side, but can wait after declarative partitioning switch)
  • update path from etc to data
  • make it run again (?)

Issue(s) this solves:

  • #812

What, in particular, needs to reviewed:

  • Theoretically the start time, interval time, date interval end are all correct so it should run at 2024-12-01.
  • reconfirm the interval time 🤔

What needs to be done by a sysadmin after this PR is merged

git pull to data_script

chmnata avatar Jan 02 '24 23:01 chmnata

As discussed, here is the template I've used within the ETL DAGs which uses generic short circuit operators short circuit operators to check the date. In the future we could wrap this all inside a generic TaskGroup which Islam has an example of in VFH!

gabrielwol avatar Jan 08 '24 16:01 gabrielwol

I think the new @task.run_if decorator in 2.10 will be helpful for simplifying EOY/EOM tasks. Something like @task.run_if(lambda context: check_jan_1st(context["ds"])) https://airflow.apache.org/docs/apache-airflow/stable/tutorial/taskflow.html#conditionally-skipping-tasks

gabrielwol avatar Aug 26 '24 16:08 gabrielwol