mozilla-pipeline-schemas
mozilla-pipeline-schemas copied to clipboard
Test BQ schema changes in CI
This is a bit of a difficult task, but the request is basically the following:
- Create BQ schema from before-commit and after-commit
- Create table in BQ with before-commit schema (
bq mk
) - Update table in BQ with after-commit schema (
bq update
) - Insert the sample data from validation directory
None of these steps should fail, and the validation data should be fully accessible.
@jklukas I think this issue is more akin to what you mentioned in #316. Long-term it would be lovely if we could do some integration tests on schema updates:
- Validate some landfill data + validation data against the JSON schema
- Generate Avro data using the ingestion Beam job
- Check the schema update (as above)
- Load the avro data into the updated BQ table
This is a long-term goal though and will eventually probably be it's own proposal. For now let's keep this as checking for BQ compatible schema updates only.