Miovision Alert: pull alert from API to our database

[chmnata]

Background:

• Miovision pulling process is in airflow

• The data validation process we have for miovision right now is check_miovision which you can read more here

• Miovision sends email alerts whenever there is an update: power outage, device unavailable, device comes back online, e.g.

  Hello Natalie Chan,

  New alert from Wellington Street West and Bay Street. Power loss began at May 23, 2023 10:45:26 AM (EDT)

  Miovision TrafficLink

• It will be nice to store this information in our database.

We can explore this Alert API first:

They do have an API for the Alerts. You can query active status on alerts at any time. http://docs.api.miovision.com/

Ideally we would have a table in the miovision_api schema with the date, time, reasoning of each outage, as well as when they come back online.

[radumas]

Playing around with the api in the docs, it looks like we can access historical data, I encoded the time as 2020-05-25T18:00:00

The API doesn't appear to validate for future timestamps however.

[gabrielwol]

I played around with the API a bit more. It only supplies a list of alerts active at time X, so to find the extent of the alerts you need to poll at some regular interval and then summarize with gaps and islands.

Here's a sample for this past Sunday, which had 8 full day alerts and one ~1 hour alert. I used a step size of 1 minute as a balance between overloading the API and wanting to identify the start/end with reasonable accuracy. It took around 2-3 minutes to run for one day.

We could also consider backfilling using CSV exports of alerts from: https://trafficlink.miovision.com/#/alerts, which look like they can be coerced into the same format as the API without too much effort and go back to 2017.