bdit_data-sources icon indicating copy to clipboard operation
bdit_data-sources copied to clipboard

Create an Airflow process to update Bluetooth detector status daily

Open radumas opened this issue 5 years ago • 1 comments

An airflow function that replaces readers_down which would run bluetooth.update_status_history(_dt date) from #250 on a daily basis and send notifications when detectors are newly down. Probably simpler to split this into three tasks in a same Airflow DAG:

  • [x] Check if there's any data for the date to notify if the Blip pipeline has failed (this should be the first task in the DAG)
  • [x] Function updates reader_locations date_last_received column
  • [x] Function inserts into bluetooth.reader_status_history from https://github.com/CityofToronto/bdit_data-sources/issues/250 whether each detector was active on that date.
  • [x] Compare bluetooth.reader_status_history to identify whether there were detectors active the day before _dt that are now not active on _dt and send a Slack notification of newly inactive detectors. Send only one Slack message with a list of failed detectors.

radumas avatar Oct 13 '20 19:10 radumas

Added the "check the pipeline worked" task to the list. I don't recommend replicating the logic of broken_readers.py because it means the table of broken readers needs to be refreshed when the pipeline is fixed. I think a simpler SELECT COUNT(*) FROM aggr_15 WHERE datetime_bin >= dt AND datetime_bin < dt + interval '1 day' would work better.

radumas avatar Jan 21 '21 15:01 radumas