squared
squared copied to clipboard
Clone Snapshots For CI Testing
Originally raised in https://github.com/meltano/squared/pull/351#issue-1336530069. Snapshots arent being fully tested if they are brand new in their isolated CI environment i.e. theres no inactivated records and its basically a "select *" vs a merge. Without this theres no historical inactivated records in snapshots during CI.
In CI we can clone the production snapshots prior to running the EL tests. That way we pull new data and snapshot that against the existing snapshot.
- add a dbt macro that clones snapshots from prod to isolated CI env
- convert CI
dbt_seed
job to more genericdbt_prep
, call clone macro from this
Other notes:
- this unlocks implementing activated dates for IP addresses. Right now we only filter for inactivated dates. Either implement or create and issue for that.
- theres a challenge here when a snapshot is new and not in production, maybe the clone script does a check and only clones if it exists. Otherwise it will fail on the first CI run.
Related to https://github.com/meltano/squared/issues/366