mozilla-pipeline-schemas icon indicating copy to clipboard operation
mozilla-pipeline-schemas copied to clipboard

Test BQ schema changes in CI

Open fbertsch opened this issue 5 years ago • 1 comments

This is a bit of a difficult task, but the request is basically the following:

  1. Create BQ schema from before-commit and after-commit
  2. Create table in BQ with before-commit schema (bq mk)
  3. Update table in BQ with after-commit schema (bq update)
  4. Insert the sample data from validation directory

None of these steps should fail, and the validation data should be fully accessible.

fbertsch avatar Apr 22 '19 16:04 fbertsch

@jklukas I think this issue is more akin to what you mentioned in #316. Long-term it would be lovely if we could do some integration tests on schema updates:

  • Validate some landfill data + validation data against the JSON schema
  • Generate Avro data using the ingestion Beam job
  • Check the schema update (as above)
  • Load the avro data into the updated BQ table

This is a long-term goal though and will eventually probably be it's own proposal. For now let's keep this as checking for BQ compatible schema updates only.

fbertsch avatar Apr 23 '19 15:04 fbertsch