aws
How to rollback to previous svc version
Hi there, in some cases, I successfully deployed a new version, but found a bug in the process of using it, and I need to quickly roll back to the previous version. I don't know how to use copilot to accomplish this.
Hello @hellof20. In Copilot we recommend users to commit your code as well as the service manifest before making a deployment. So you can quickly rollback to the previous version by checking into the previous commit and deploy again, since all the setting for the service should be included in the manifest as well as in your code.
Hi @iamhopaul123 , checking into the previous commit and deploy again, it will takes lots of time. I know that every deploy will generate a new ECS service task definition, but I found that the previous task definition status has become inactive, and I cannot manually modify the task definition version of the service. I think if there is something like copilot svc deploy --rollback it will be very helpful to solve the problem.
To emphasize importance of this feature, I wanted to add issue we saw.
Due to AWS SSM Service issue that came up during our release, only partial services were deployed via code-pipelines during our release process. Our code pipelines are deploying to 4 envs (and this occurred during deployment to last env, PROD 😖)
Its not easy to fix this situation, as we will need to revert all dependent service code, and promote them all the way through all envs OR manually deploy.. Basically not easy. Of course, core issue is services not having backward compatibility among themselves :)
In codepipeline, if we have an option to revert a deployment to previous build, we can trigger this and revert all services easily. I think this is an important piece, especially due to lack of blue green deployment support.
Hello @gautam-nutalapati !
Thanks for the input! They make sense to me. I just have a follow up question for me to start understanding how to make it helpful:
Is your pipeline triggered by some push activities to your code repository? What would prevent you to just rollback your code (assuming that includes the application code and the manifests) - which would then trigger the pipeline to deploy an older version? Was it because the CFN (for example during the DeployToProd stage)was stuck in an un-updatable state?
Basically my concern is that what made your manual rollback difficult could also make the automatic rollback faulty 🤔. I'd like to understand the issue better so that if/when we do get to this feature, we could approach it in a helpful way.
There was no blocker for manual rollback. Pipelines are triggered by push activity. CFN stack were in good rolled-back state.
Services have to be promoted via underlying envs in pipeline. [Tests running after deploy + CFN deploy] time makes it slow to promote code revert (Can't wait that long to revert PROD). For these reasons, I didn't prefer rollback from code. Rolling back ECS task OR manual deployment was faster in this case.
Some additional thoughts:
For issues like SSM service is down, which cause out-of-sync svc deployments, we cannot deploy anything via CFN even when code's reverted. Task definitions in ECS can be reverted, but there now disconnect as add-ons are out of sync.
If there was an issue thats not blocking deployment, e.g. service related, we can rollback via CFN, but reverting add-ons containing resources that can't be reverted (like DDB) can bring other kinds of issues.
Personally, due to above thoughts, I take back my prev comment, I would vote for blue-green feature over reverts feature.