bdit_data-sources icon indicating copy to clipboard operation
bdit_data-sources copied to clipboard

Published 20 hours ago • verified publisher icon CityofToronto

Automate end of year maintenance

Open radumas opened this issue 3 years ago • 5 comments

We have partitioned tables that are monthly/annually. We need to create new ones for the new year and update triggers etc. Procedure in #251, hopefully as much as possible can be made into a DAG.

  • [x] Here traffic analytics
  • [x] Bluetooth
  • [x] Miovision
  • [x] Holidays table
  • [x] WYS table

radumas avatar Dec 30 '21 17:12 radumas

Super pleased to have discovered how to use xcom to get a return value from a function and use it in a Slack message for updating the HERE TRIGGER function. Did realize after all that that I could instead use the loops in Python to recreate the long logic chain each year for every year and month from next year to 2012.

Also ran into this baffling bug https://stackoverflow.com/questions/70538612/tables-created-via-function-not-recognized-when-trying-to-update-trigger-functio

radumas avatar Dec 31 '21 03:12 radumas

Well got the opportunity to do just that with Bluetooth. Python is ready just need to add the 2 tasks to the DAG.

radumas avatar Dec 31 '21 03:12 radumas

Bluetooth works. Did have to grant CREATE permissions on the bluetooth schema to bt_bot but I don't suuuuper see why that would be an issue.

radumas avatar Dec 31 '21 14:12 radumas

Holidays ✅ Might want to revisit the holidays package since it's reporting that New Year's Day is observed 2021-12-31...

radumas avatar Dec 31 '21 18:12 radumas

WYS ✅

There should prooooobably be more documentation about these changes somewhere.

radumas avatar Dec 31 '21 19:12 radumas