httparchive.org
httparchive.org copied to clipboard
Update report JSON exports to `all` dataset
For the script located in https://github.com/HTTPArchive/bigquery/tree/master/sql:
- rewrite queries to a new stable schema
- move the script to another currently used repo where it fits the most
- trigger this step automatically upon successful BQ pipeline completion
I can help with SQL rewrite, but not familiar with nuances of these bash scripts
Related issues:
- documentation https://github.com/HTTPArchive/data-pipeline/issues/21
- alerts for failed reports (already implemented for failed workflows on Dataform) https://github.com/HTTPArchive/data-pipeline/issues/24