Miovision - add Metrolinx Jane and Finch intersection

[cczhu]

Miovision has a contract with Metrolinx to provide TMC volumes at Finch Avenue West and Jane Street (there's a few other locations listed by the API, but these are only TrafficLink locations, as Jesse mentions below). We have been given an API key (e-mail from Jesse to Aakash October 15, 2020) to access this feed. Pulling from this API should be identical to using our own; however based on how our pipeline is written, we'll need to make the following modifications:

1. intersection_tmc.py reads the Miovision API key by reading in a file with configparser. To switch APIs, the simple thing to do is to run intersection_tmc.py twice, switching API key files the second time.
2. intersection_tmc.py finds locations by reading miovision_api.intersections. Simply adding the Jane and Finch intersection to miovision_api.intersections may lead to "error": "Intersection not found" when pulling from the API, which needs to be properly handled in intersection_tmc.py. Alternatively, we could have a separate set of tables that handle pulling from Metrolinx.
3. When aggregating to the 15-minute level, miovision_api.intersections, miovision_api.intersection_movements and other tables that hardcode intersection UIDs are used. In some cases, these hardcoded intersection UIDs are used to populate zero-padded aggregated volumes, so it's not possible to aggregate TS and Metrolinx intersections separately without producing a bunch of zero-volume aggregated values.

One possible design is to have to completely separate data streams - one for TS Miovision data, and one for Metrolinx. The major downside of this is any time the pipeline is updated it'll have to be mirrored for both data streams, which is a recipe for disasterous software development.

A more reasonable pipeline design is this:

• Add a API source column to miovision_api.intersections. This will be used by intersection_tmc.py but not by aggregation scripts, which will be agnostic to where the data comes from.
• Modify intersection_tmc.py to do the following:
  • Read and store both API keys, possibly in separate files
  • When it comes time to pull from the API, use the appropriate API key depending on API source. Since the API key is passed for every data pull request the script makes, this doesn't add any additional work to intersection_tmc.py.
• Data sitting in miovision_api.volumes is API-agnostic, as is the pipeline to aggregate volumes up to 15-minute bins.

The second step should become much easier once #407 is merged to production and we start using the MiovTMCPuller class. I mentioned in the development Jupyter notebooks to #403 that MiovTMCPuller could eventually incorporate the class that stores intersection information. This would not only simplify the code, but if MiovTMCPuller also had two API keys stored it could easily switch between them depending its stored intersection information.

[jwrcoleman]

They only have a smartsense unit at Jane & Finch. The others are just trafficlink devices.

[cczhu]

Thanks Jesse, updated description.

[gabrielwol]

@jwrcoleman can we close this as not planned?

[jwrcoleman]

I think it is actually still planned, we can confirm with them.... just haven't done anything about this for a long time. But its a very important locations for work around the Jane RapidTO