rds-snapshot-export-to-s3-pipeline
rds-snapshot-export-to-s3-pipeline copied to clipboard
Output the snapshot to set location?
So we're using this to export a couple of RDS DBs into our data lake. The problem I have is each day the snapshot is created in a new folder (with the new date) so my Glue Crawlers create another set of tables with the latest data. This means our analysts have no easy way to find the latest data - they see multiple copies of "my_table" with a random string suffix.
What I'd like to do is always output to one set location for my snapshot, and effectively overwrite the data each day. That way my analysts will always be looking at the latest version of the data. The only way I can see this working is to output the snapshots to a "staging" type area, and have some sort of job that moves the latest data each morning. Has anyone found a better way to do this?