fst icon indicating copy to clipboard operation
fst copied to clipboard

get a data diff between dev and prod

Open sungchun12 opened this issue 2 years ago • 1 comments

Research

https://medium.com/snowflake/minimalist-snowflake-table-compare-using-data-diff-ba67cc4f904c

  • look at audit helper because datadiff has a lot of rough edges to it
  • use dbt 1.5+ to run invocations
  • update compare to prod chart to update y axis for row counts, bytes, other stats
  • have audit helper do hot reloading in the logs and store invocations automatically instead of a dropdown UI: https://hub.getdbt.com/dbt-labs/audit_helper/latest/
  • I'll probably have only one of these functions for the sake of MVP
  • compare to dev to prod only. That's where it's most useful: compare_relation_columns
  • set defaults for file generation, look at profiles.yml for schema info
  • store historical audit reports for each iteration

sungchun12 avatar May 24 '23 16:05 sungchun12

This is a stretch goal. I'm okay to ship without this.

sungchun12 avatar May 24 '23 16:05 sungchun12