bdit_data-sources
bdit_data-sources copied to clipboard
Create an Airflow process to update Bluetooth detector status daily
An airflow function that replaces readers_down which would run bluetooth.update_status_history(_dt date) from #250 on a daily basis and send notifications when detectors are newly down. Probably simpler to split this into three tasks in a same Airflow DAG:
- [x] Check if there's any data for the date to notify if the Blip pipeline has failed (this should be the first task in the DAG)
- [x] Function updates
reader_locationsdate_last_receivedcolumn - [x] Function inserts into
bluetooth.reader_status_historyfrom https://github.com/CityofToronto/bdit_data-sources/issues/250 whether each detector was active on that date. - [x] Compare
bluetooth.reader_status_historyto identify whether there were detectors active the day before_dtthat are now not active on_dtand send a Slack notification of newly inactive detectors. Send only one Slack message with a list of failed detectors.
Added the "check the pipeline worked" task to the list. I don't recommend replicating the logic of broken_readers.py because it means the table of broken readers needs to be refreshed when the pipeline is fixed. I think a simpler SELECT COUNT(*) FROM aggr_15 WHERE datetime_bin >= dt AND datetime_bin < dt + interval '1 day' would work better.