bdit_data-sources icon indicating copy to clipboard operation
bdit_data-sources copied to clipboard

Miovision Aggregation Improvements (ATR view, intersection_movements AFTER INSERT trigger, unacceptable_gaps)

Open gabrielwol opened this issue 7 months ago • 1 comments

What this pull request accomplishes:

  1. Remove miovision_api.volumes_15min (table) in place of view (volumes_15min_atr_filtered). Reduces duplicate storage (~75GB) and complexity in aggregation.
  2. Add an AFTER INSERT trigger to intersection_movements to help aggregate and pad new movements into volumes_15min_mvt.
    • volumes/miovision/sql/function/function-add_intersection_movement_padding_values.sql: based on existing inserts into volumes_15min_mvt, except using new_rows (transition relation) instead of intersection_movements and without date limits.
  3. Move unacceptable_gap removal from table inserts to filtered view: less complexity in adding and removing unacceptable_gaps.

Summary of files changed: Many files have minor changes including renamed references, removal of ATR aggregation, big fluff (intersection_movements, volumes_15min_tmc).

Files with major changes:

Issue(s) this solves:

  • Closes #868, #879: filters unacceptable_gaps via view
  • Progress towards closing issues around intersection_movements: #406, #855

What, in particular, needs to reviewed:

  • [ ] Another option considered is adding movement map columns entry_dir, exit_leg, exit_dir to TMC table so ATR view wouldn't need to join to movement_map. Downside: more confusing TMC table, does not match current TMC table.
  • [ ] Ok to anti join instead of set unacceptable_gaps to null in filtered view?
  • [ ] New naming: match existing to reduce dependency changes or make contents more explicit?
Old New
TMC volumes_15min_mvt volumes_15min_mvt_unfiltered
volumes_15min_mvt_filtered volumes_15min_mvt_filtered
ATR volumes_15min
volumes_15min_filtered volumes_15min_atr_filtered

What needs to be done by a sysadmin after this PR is merged

E.g.: these tables need to be migrated/created in the production schema.

gabrielwol avatar Jul 24 '24 21:07 gabrielwol