Add ability to re-use existing API Gateway with deploy

[surfkansas]

Currently, the only way to re-use an existing API Gateway endpoint in a chalice deploy command is to have the deployed resource index stored in the .chalice/deployed/<stage>.json file. In a multiple user environment, this works fine as long as the <stage>.json file is committed to the GIT repo.

However, when trying to use chalice deploy in a CI/CD pipeline (such as GitLab CI), the first release must commit the new <stage>.json file to the repository. Otherwise, each deployment creates a new API Gateway endpoint. Other resources, such as Lambda function or IAM role, which have a consistent name per account and stage, are correctly re-deployed.

Proposal:

Add an additional option, --reuse-api-gateway to the chalice deploy command. Usage would look something like:

chalice deploy --reuse-api-gateway --stage prod

Expected behavior without option:

Chalice would deploy exactly as it currently does

Expected behavior with option:

Chalice would interrogate the API Gateway to find the API ID that matches the name of the api being deployed and re-use it, rather than creating a new one.

[stealthycoin]

I wouldn't really recommend using chalice deploy in a CI/CD environment. Thats more what chalice package is for which removes this whole issue. Is there a reason you can't use package instead for your CI/CD pipeline?

[surfkansas]

There are a few reasons that chalice deploy works better.

First, it is available "out of the box". The time it takes to set up the CI/CD for multiple chalice services is non-trivial. With chalice deploy, everything can be packaged up into a very simple "templated" CI/CD pipeline.

Second, it is easier to support with CI/CD pipelines that run on non-AWS platforms. Using a different CI/CD process (such as GitLab CI) means that the out of the box Chalice CI must in many places be hand-recreated.

Some resources, such as S3 subscriptions, cannot be included in a Cloud Formation template.

[markalexander]

Some resources, such as S3 subscriptions, cannot be included in a Cloud Formation template.

Yes, this is exactly why I use chalice deploy for some pipelines. It's a pain.

Two possible workarounds I can see:

• Commit the deployment .json back to the same repo as part of the pipeline (as you noted). Need to add logic to your pipeline to ignore these commits. Or do the first one manually and assume you don't change anything relevant in there.
• Run some code to re-associate your request router (e.g. CloudFront) with the new API gateway endpoint. But the default deploy already comes with Edge Optimization, so you need to disable that too. In my experience this leads to downtime as CloudFront switches over. And you need to delete the old stack.

Pretty ugly either way. Maybe someone has a better suggestion.

[wollerman]

Alternatively, what about a backend type extension that specifies where the deployed.json can be found? Similar to how terraform can use a s3 remote backend. This way, you can specify a backend parameter in the config.json which points to a particular s3 location and allows you to deploy that configuration. e.g.

{
"stages": {
    "dev": {
      "api_gateway_stage": "api",
      "backend": {
        "location": "s3",
        "bucket": "mybucket",
        "key": "path/to/my/key",
        "region": "us-west-2",
     }
   }
  }
}

-or-

{
"stages": {
    "dev": {
      "api_gateway_stage": "api",
      "backend": {
        "location": "local"
     }
   }
  }
}

As a reference, here's terraforms documentation: https://www.terraform.io/docs/backends/types/s3.html

This would mean the configuration from any client is stored in a global location accessible to anything from other teammates to CI/CD jobs.

[kyleknap]

@wollerman I like your idea. We also have a GitHub issue tracking having the deployed.json store in S3 for redeployment/sharing purposes: https://github.com/aws/chalice/issues/1027.

Marking this specific issue though as a feature request for having documentation and tooling for creating CI/CD pipelines that use chalice deploy instead of chalice package as there are advantages of using chalice deploy instead of chalice package.

[subssn21]

This issue not only occurs in API Gateway, It also happens with SQS Queues as well.

The first time the deploy works and all subsequent times it will fail unless you have updated the deployed files in .chalice.

An error occurred (ResourceConflictException) when calling the CreateEventSourceMapping operation: An event source mapping with SQS arn (" arn:aws:sqs:us-XXXX-X:XXXXXXXXXXXXX:XXXXXXXX ") and function (" XXXXXXX ") already exists. Please update or delete the existing mapping with UUID XXXXXXXX-XXXX-XXXX-XXXX-XXXXXXXXXXXX

[ibmboy19]

Please provide the reuse existing api gateway feature. Thanks!

[davidolmo]

Definitely looking into the possibility of reusing the API Gateway

[Zuiluj]

Is this feature now available or there is a workaround for it?

[joedevgee]

This is a critical feature for teams working in non-AWS CI/CD environment (i.e. Travis, CircleCI, GithubAction). Is someone actively working on this?

[gerritjvv]

To help others that encounter the same issue. You MUST commit and push the .chalice/deployed/ folder contents after a deploy.

That means, first deploy your application, the commit the new changes and push.

In bitbucket you can use:

- chalice deploy
- git add --force .chalice/deployed
- git commit -a --allow-empty -a -m "[skip ci] pushing deployment history"
- git push origin HEAD