squared icon indicating copy to clipboard operation
squared copied to clipboard

Clone Snapshots For CI Testing

Open pnadolny13 opened this issue 2 years ago • 1 comments

Originally raised in https://github.com/meltano/squared/pull/351#issue-1336530069. Snapshots arent being fully tested if they are brand new in their isolated CI environment i.e. theres no inactivated records and its basically a "select *" vs a merge. Without this theres no historical inactivated records in snapshots during CI.

In CI we can clone the production snapshots prior to running the EL tests. That way we pull new data and snapshot that against the existing snapshot.

  • add a dbt macro that clones snapshots from prod to isolated CI env
  • convert CI dbt_seed job to more generic dbt_prep, call clone macro from this

Other notes:

  • this unlocks implementing activated dates for IP addresses. Right now we only filter for inactivated dates. Either implement or create and issue for that.
  • theres a challenge here when a snapshot is new and not in production, maybe the clone script does a check and only clones if it exists. Otherwise it will fail on the first CI run.

pnadolny13 avatar Aug 12 '22 15:08 pnadolny13

Related to https://github.com/meltano/squared/issues/366

pnadolny13 avatar Aug 23 '22 14:08 pnadolny13