Post-deployment actions for pipeline

[Lou1415926]

People might need to run some post-deployment scripts that are not considered as tests, or should be run only after the test succeeds. Copilot could support a post-deployment action that does this, backed by CodeBuild. This action is similar to the existing TestCommands action, but is different in that its buildspec should be customizable. For example, user could provide a buildspec_after.yml for this action.

[matthewhembree]

Adding generic pre/post hooking to stages/jobs would be nice. Pre/post for deployment. Certainly pre/post for tests (e.g. pre_test to run migrations).

[DrewVartanian]

+1

[efekarakus]

Hi folks! Thank you so much for all the 👍 on this issue. For clarification purposes, would you mind providing some additional info on how you'd like to use this feature?

• Would you like to run the post_deployment action after all your services in the pipeline get deployed or would you like to have a separate action per deployed microservice?
• Do the post_deployment actions need to access resources in your environment's VPC? Or can it be outside of the VPC?
• What are some typical commands you'd like to run part of the build project? For example, how would you do db migrations?

[DrewVartanian]

Hi @efekarakus . I'm looking to run a PHP Doctrine migration for a Symfony project, with a command like php app/console doctrine:migrations:migrate from within the docker container (only need to do this on one of the containers, as they share a database). Ideally, this would be run after the uploads to ECR, and directly before the deployments to ECS/Fargate. In our old setup, where we had projects on an EC2 box, we would checkout and build the new version of the codebase, and run migrations as the last step before switching from the old build to the new build. I found this post (https://sagargiri.com/build-aws-codepipeline-for-cicd) which talks about using an appspec.yml file for BeforeInstall and AfterInstall hooks when using CodePipeline through the AWS console, but I've had trouble finding any documentation on how to do a migration when using Pipeline with Copilot, or how to best run a migration that needs to be run from a built image.

[miketheman]

Hey @efekarakus ! Found this one today, and wanted to provide some more color, from my perspective:

• Would you like to run the post_deployment action after all your services in the pipeline get deployed or would you like to have a separate action per deployed microservice?

In a perfect world, we split items like migrations from the actual code that would reference them, but that rarely actually happens. With a data migration such as adding a column to a database table, I'd want that operation to run before any of the code that references it is active.

With other migrations like a column removal, I likely have to split the migrations from the code, and ship the code that removes the references to the column before I ship another time with the migration that drops the column, since I can never guarantee that the running code won't reference the removed column.

For a post_deployment action, I consider the entire deployment a single unit of operation, so no matter how many services there are, only one pre_ and post_deployment event makes sense to me. You either deploy or you don't - unless the microservices won't roll back atomically? Can that happen today?

• Do the post_deployment actions need to access resources in your environment's VPC? Or can it be outside of the VPC?

With data migrations, yes, since it runs an instance of the application container that has the new code and migrations, and execute against the same resources, using the same variables like DATABASE_URL, etc For other operations like posting a web call of "I'm done!", it'll need access to whomever consumes that web call.

• What are some typical commands you'd like to run part of the build project? For example, how would you do db migrations?

A pre_deployment (post build!) step might be python3 manage.py migrate or rails db:migrate (or the Symfony command Drew supplied, etc), and if successful, start replacing application container instances and complete the deployment.

Hope this all makes sense, feel free to let me know if I can help clarify anything!

[efekarakus]

Hi @miketheman !! Apologies on the delayed response, we're getting ready to ship v1.22 so it's been busy few days :)

Thank you so much for the detailed response, this helps a ton. I understand the need of having a pre_, and post_ option depending on the type of update 👍

So it sounds like there will be moments when the pre_deployments_buildspec.yml will get updated and sometimes the post_deployments_buildspec.yml. Am I capturing accurately?

I assume that the scenario of a successful ✅ pre_deployment and then a failed ❌ deployments is not to be worried about because in the pre_deployment we shouldn't delete any columns or tables such that a deployment rollback fails.

[miketheman]

Hey @efekarakus ! No worries, Good luck on 1.22 😉

it sounds like there will be moments when the pre_deployments_buildspec.yml will get updated and sometimes the post_deployments_buildspec.yml

I think so, largely depending on how the end user wants to structure their deploy process.

I can imagine a deploy process that does this (I do this today with homegrown orchestrations):

• pre_deployment: send signal to tracking system "deployment #N starting", and then "run an instance of the new container image to run migration commands" (since the new image will contain the new code)
• deployments ....
• post_deployment: send signal to tracking system: "deployment #N complete"

If during the deployments phase something goes wrong, then post_deployment would never execute. It does raise a question of whether the "entire deployment" process should be atomic, and if so, then there's utility in the post_deployment side to execute something like on_failed_deploy: execute database rollback command... to restore deleted columns. Unlikely, and I don't think I've ever done that specific kind of database rollback - it comes down to "how atomic do you want your database state to be" and that's a tough question to answer. A good recent post on the topic that expresses some of the challenges: https://cheigh.me/blog/posts/2022-05-10-enforcing-zero-downtime-django-migrations.html (we don't follow precisely their solution).

In our homegrown system, we allow the deployer to select before and after migration strategies, but that's caused some incidents when the deployer is unfamiliar with them, so have started to encourage before only.

[efekarakus]

This is great, thanks @miketheman !

[spelka]

Hello,

one reason i would like to be able to run a CodeBuild action later in our pipeline is that we have a set of 'end-to-end' integration tests which test across multiple applications/services.

We do our unit-testing inside the Build step.

But, the idea is that we need to have the fargate application FIRST deployed into a QA/Test environment first, and THEN run the end-to-end test to see if our new version of the application plays nice with the rest of the system. Running it in the build stage wouldn't actually tell you anything, since you are (at that point in time) still running the old version of the software.

I also like the idea of submitting a buildspec-type file for this, becuase all your secret access is handled neatly inside the buildspec.

One thing of note, I do use secrets inside my test invocation (my integrators gave me credentials to use against their endpoints), so it would be nice if copilot also could eventually also handle secret creation in secret manage as a part of the workflow? although that might be expanding the scope of this story beyond its original parameters. I just added some food for thought.

[lennertcc]

• Would you like to run the post_deployment action after all your services in the pipeline get deployed or would you like to have a separate action per deployed microservice

I'd say both. I would want to write the following pipeline manifest:

stages:
  - name: prod
    require_approval: true
    pre_deployment: copilot task run -n db-migrate --env prod
    post_deployment: <some task>
    deployments:
      backend:
      frontend:
        depends_on: [backend]
        pre_deployment: <some task>
        post_deployment: copilot task run -n collect-statics --env prod

Specific deployment commands in this example are just a thought, anything would be fine as long as I can run commands using the container images in ECR.

This case could also be solved by making the db-migrate action a pre_deployment step of the backend. So if I had to choose, I would say a separate action per deployed microservice.

[Lou1415926]

Related: #2428, https://github.com/aws/copilot-cli/issues/4497#issuecomment-1437511104, #4372

[spelka]

One more item to add to the conversation, i think it would be good to have the option to add an arbitrary number of pre-build or post-build steps for CodeBuild.

Vulnerability Scans, CodeCoverage reports, these are things on my to-do list. However, I'm not sure that the intent of a pipeline is to jam all this into one stage / action. From the examples I have seen, it looks like a lot of these discrete processes are packaged into separate stages, or at the very least separate actions within a stage.

I could be wrong, since I am pretty new to pipeline management. Can someone weigh in on this?

[opusmagnum]

What about a more generic approach to extend the (by Copilot) provided configuration with the combination of: the pipeline ... package and pipeline ... extend (with CDK)? Like it is already released for svc and env. In this case -- all the above requested scenarios could be implemented by users themself -- with adding configuration parameters or new resources (e.g. additional CodeBuild projects, etc.).

[j-krl]

Any update on this? Still no great way to run migrations in a pipeline

[KollaAdithya]

Hey @j-krl ! we haven't started working on post_deployments yet 🙇 . Please upvote the issue to help us priortize! For database migrations you can use test_commands as pointed in this workaround https://github.com/aws/copilot-cli/issues/2992#issuecomment-964654705, even though it is not an ideal solution to have db migrations with test_commands.

[huanjani]

@opusmagnum, just in case you missed it, Copilot's pipeline overrides feature was recently released! See https://aws.github.io/copilot-cli/blogs/release-v129/#pipeline-overrides. (While we do call pipeline package under the hood, it isn't a user command yet.) Thanks for the comment!

[dannyrandall]

Hey everyone! We just released this feature in v1.30.0!🚀🎉 You can take a look at the blog post that covers how you can add pre_deployments and post_deployments to your pipelines!