Feature request: Allow Aurora storage init to create cluster from existing snapshot id

[jasonmarlin]

It would be a useful feature if we could use copilot to create db clusters from existing snapshots.

Example: I have a legacy application that I would like to transition to copilot. I can easily spin up a new environment, including Aurora Serverless cluster using a few copilot commands. However, there is no simple way to "hydrate" this new database. Using utilities that rely on SQL export/import will take a very long time and require some level coding to automate. Since the standard aws cli commands allow for cluster creation from snapshot, it would be great if we could simply inherit that for copilot.

Another useful scenario would be the case where you'd like to quickly clone your production environment to a temporary dev setup, including data. You'd simply create or reference the latest snapshot and be ready to go.

Something like: copilot storage init -t Aurora --restore-snapshot-id xxxxyyyyy [other params]

Thank you!

[efekarakus]

That's a great idea @jasonmarlin ! Thank you for the feature request, it makes sense to me.

It seems to be related to https://github.com/aws/copilot-cli/issues/2947

[kwood]

Another use case for this is disaster recovery — as it stands, while snapshots are being taken of my Aurora cluster automatically, I can't see a way to roll back to a previous database snapshot.

From the AWS console we can only restore a snapshot to a new database instance. Since Copilot manages the database credentials, I'm not sure how I'd restore a db snapshot in an emergency.

[kwood]

This is also useful in developer workflows — for example, right now I'd like to roll back my staging database to a prior state so that I can re-test a database migration against it. I thought this would be fairly trivial but it doesn't appear to be.

[efekarakus]

Hi @kwood !

Sorry for the inexperienced question around RDS, would you mind explaining what is the shortcoming of running the following command?

aws rds restore-db-cluster-from-snapshot \
    --db-cluster-identifier mynewdbcluster \
    --snapshot-identifier mydbclustersnapshot \
    --engine aurora|aurora-mysql|aurora-postgresql

(https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/aurora-restore-snapshot.html)

Does that achieve the desired outcome but you'd like to do through Copilot instead?

[kwood]

@efekarakus if I understand correctly, that creates a new db cluster based on the snapshot — it does not configure security groups or secrets such that my application can use it, and it doesn't do anything with the old cluster.

Were you to just run that command, I believe you'd have to:

1. Configure security groups to enable access to the new DB cluster
2. Manually change the DB secrets in your secret store
3. Restart all the containers of your application

I'm not sure but I suspect might be a (4), which is to do something to ensure Copilot doesn't undo any of that work on the next deploy

I could be wrong about one or more of those steps — please correct me if I am!

So I think the desired outcome is that with a single copilot command I could select a DB snapshot to roll an environment back to, which would perform all the necessary steps.

[cool-raj]

Any Progress on this FR ?? I am also looking for a solution for this exact issue ?

[KollaAdithya]

Sorry @cool-raj to inform that we have any updates to share on this as of now 😞

[bathsundeep-graticule]

@kwood We also wanted to add disaster recovery for RDS and came to this working solution. The snapshot-identifier is a close solution, but to maintain the security groups, secrets, etc of the copilot environment with your RDS cluster, you'll need to deploy the cluster with copilot. To do so:

1. Run copilot svc package --output-dir . to get the Cloudformation ymls that copilot generates for your stack & addons. Our RDS cluster is part of a copilot addon which contains the related subnets, security groups, and secrets.

2. That alone should give us something you could deploy with copilot, but to add snapshot restoration we just need to add one key. The RDS cluster is found with Type: 'AWS::RDS::DBCluster' and we add a new Property SnapshotIdentifier: <Manual_Snapshot_ARN>. This key gives the cluster a snapshot to bootstrap from. Adding, removing, or updating this key to a Cluster will destroy and restore the cluster again. And don't worry, consecutive deployments will not restore as long as the SnapshotIdentifier doesn't change. Cloudformation docs

3. Further, we can control the snapshots for different deployment environments with a mapping in the Cloudformation template.

Mappings:
  mydbclusterSnapshotsMap:
    dev:
      ARN: 'arn:aws:rds:...'
    test:
      ARN: 'arn:aws:rds:...'
    beta:
      ARN: 'arn:aws:rds:...'
    prod:
      ARN: 'arn:aws:rds:...'

And making the property SnapshotIdentifier: !FindInMap [mydbclusterSnapshotsMap, !Ref Env, ARN]

It's important to note that you'll need to use a MANUAL snapshot's ARN for this to work. You can take a system snapshot and manually copy it to create a long living snapshot to recover from that will survive any cluster going down.

4. Make your modified RDS Cloudformation template a new addon that lives next to your service manifest, so it can all be deployed together with copilot svc deploy.

I think this would accomplish everything you'd need, though I'm not sure if service manifest containers would be guaranteed to restart if the addon manifest is the only thing changing..

[iamhopaul123]

though I'm not sure if service manifest containers would be guaranteed to restart if the addon manifest is the only thing changing..

I don't think Copilot will necessarily run new tasks unless you do copilot svc deploy --force. However, there'll be CFN stack update for sure.