httparchive.org icon indicating copy to clipboard operation
httparchive.org copied to clipboard

Update report JSON exports to `all` dataset

Open max-ostapenko opened this issue 4 months ago • 14 comments

For the script located in https://github.com/HTTPArchive/bigquery/tree/master/sql:

  • rewrite queries to a new stable schema
  • move the script to another currently used repo where it fits the most
  • trigger this step automatically upon successful BQ pipeline completion

I can help with SQL rewrite, but not familiar with nuances of these bash scripts

Related issues:

  • documentation https://github.com/HTTPArchive/data-pipeline/issues/21
  • alerts for failed reports (already implemented for failed workflows on Dataform) https://github.com/HTTPArchive/data-pipeline/issues/24

max-ostapenko avatar Oct 16 '24 13:10 max-ostapenko