bdit_data-sources
bdit_data-sources copied to clipboard
812 eoy table
What this pull request accomplishes:
- taskflowify-ish the dag (definitely more things can be done on the bluetooth side, but can wait after declarative partitioning switch)
- update path from etc to data
- make it run again (?)
Issue(s) this solves:
- #812
What, in particular, needs to reviewed:
- Theoretically the start time, interval time, date interval end are all correct so it should run at 2024-12-01.
- reconfirm the interval time 🤔
What needs to be done by a sysadmin after this PR is merged
git pull to data_script
As discussed, here is the template I've used within the ETL DAGs which uses generic short circuit operators short circuit operators to check the date. In the future we could wrap this all inside a generic TaskGroup which Islam has an example of in VFH!
I think the new @task.run_if decorator in 2.10 will be helpful for simplifying EOY/EOM tasks. Something like
@task.run_if(lambda context: check_jan_1st(context["ds"]))
https://airflow.apache.org/docs/apache-airflow/stable/tutorial/taskflow.html#conditionally-skipping-tasks