copilot-cli
copilot-cli copied to clipboard
aws
HandlerErrorCode: GeneralServiceException when trying to deploy rails app
Hi,
I'm trying to deploy a rails app using AWS copilot. I keep encountering this error during copilot deploy of my front-end service - "HandlerErrorCode: GeneralServiceException" error. I've attempted to view deploy logs but can't find the logs for this. I've already tried copilot svc logs but this doesn't show logs for deployment process. I also couldn't find the deploy logs in aws management console in 'ecs' section.
Also, I'm running docker on my Mac M1 so I updated manifest file with "platform: linux/arm64". I saw other issues related to this but this didn't fix the problem.
Questions
- Where can I find logs for deploy of front end app?
- Can you tell me anything I'm missing for deploying a rails app using aws coplilot?
shell output doing copilot deploy
(base) ➜ webapp git:(main) ✗ copilot deploy
Name: front-end
Only found one environment, defaulting to: production
Environment production is already on the latest version v1.8.0, skip upgrade.
Building your container image: docker build -t ***.dkr.ecr.us-east-2.amazonaws.com/webapp/front-end --platform linux/arm64 /Users/user/Dropbox/programs/webapp -f /Users/user/Dropbox/programs/webapp/prod.Dockerfile
[+] Building 50.6s (19/19) FINISHED
=> [internal] load build definition from prod.Dockerfile 0.0s
=> => transferring dockerfile: 951B 0.0s
=> [internal] load .dockerignore 0.0s
=> => transferring context: 2B 0.0s
=> [internal] load metadata for docker.io/library/ruby:3.0 0.3s
=> [ 1/14] FROM docker.io/library/ruby:3.0@sha256:7c57b474163e01f1518ff830dffef023fbd014378edd414526562137edc1400f 0.0s
=> => resolve docker.io/library/ruby:3.0@sha256:7c57b474163e01f1518ff830dffef023fbd014378edd414526562137edc1400f 0.0s
=> [internal] load build context 3.8s
=> => transferring context: 32.36MB 3.6s
=> CACHED [ 2/14] RUN curl https://deb.nodesource.com/setup_12.x | bash 0.0s
=> CACHED [ 3/14] RUN curl https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add - 0.0s
=> CACHED [ 4/14] RUN echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list 0.0s
=> CACHED [ 5/14] RUN apt-get update -qq && apt-get install -y nodejs postgresql-client yarn 0.0s
=> CACHED [ 6/14] WORKDIR /webapp 0.0s
=> [ 7/14] COPY . /webapp 10.1s
=> [ 8/14] RUN bundle install 17.1s
=> [ 9/14] COPY entrypoint.sh /usr/bin/ 0.0s
=> [10/14] RUN chmod +x /usr/bin/entrypoint.sh 0.3s
=> [11/14] RUN ./bin/bundle add jsbundling-rails 0.4s
=> [12/14] RUN ./bin/rails javascript:install:esbuild 8.6s
=> [13/14] RUN ./bin/rails css:install:sass 3.6s
=> [14/14] RUN rake assets:precompile 3.1s
=> exporting to image 3.0s
=> => exporting layers 3.0s
=> => writing image sha256:2eebafda0c85bad623c13134dccce1b5f1d41eb19a0befb1ae40effa2c946c1d 0.0s
=> => naming to ***.dkr.ecr.us-east-2.amazonaws.com/webapp/front-end 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Login Succeeded
Logging in with your password grants your terminal complete access to your account.
For better security, log in with a limited-privilege personal access token. Learn more at https://docs.docker.com/go/access-tokens/
Using default tag: latest
The push refers to repository [***.dkr.ecr.us-east-2.amazonaws.com/webapp/front-end]
d32d3164d4ff: Pushed
bd01b1db218b: Pushed
3622f4d0249c: Pushed
15ac93e6f83b: Pushed
06cc5a163598: Pushed
b09eb0127cdb: Pushed
cfd4ebd79bd7: Pushed
6bdcf7ae539c: Pushed
58e7a52b9c1b: Pushed
cad91ab00f81: Pushed
6daad5dd39f8: Pushed
09780ccc5e32: Pushed
d2d95ffb6a85: Pushed
1d15e266d169: Layer already exists
a84e803462b4: Layer already exists
b29575bcee7c: Layer already exists
b80781e7a3a9: Layer already exists
f523640755f1: Layer already exists
a6683c53303c: Layer already exists
98cc55516aa8: Layer already exists
63c098a71e7b: Layer already exists
latest: digest: sha256:ebfa86f2ea84b71fe619b5fd37de981012834d930b3f0a1deb2f1596a9795eb8 size: 4732
✔ Proposing infrastructure changes for stack webapp-production-front-end
- Creating the infrastructure for stack webapp-production-front-end [rollback complete] [1902.3s]
The following resource(s) failed to create: [Service]. Rollback reques
ted by user.
- Service discovery for your services to communicate within the VPC [delete complete] [1.2s]
- Update your environment's shared resources [update complete] [178.7s]
- A security group for your load balancer allowing HTTP and HTTPS traffic [create complete] [6.5s]
- An Application Load Balancer to distribute public traffic to your services [create complete] [154.0s]
- An IAM Role for the Fargate agent to make AWS API calls on your behalf [delete complete] [1.2s]
- A CloudWatch log group to hold your service logs [delete complete] [1.2s]
- An ECS service to run and maintain your tasks in the environment cluster [delete complete] [96.5s]
Resource handler returned message: "Error occurred during operation 'E
CS Deployment Circuit Breaker was triggered'." (RequestToken: 00c3596a
-44d4-4516-45bd-07dfa61254bf, HandlerErrorCode: GeneralServiceExceptio
n)
Deployments
Revision Rollout Desired Running Failed Pending
PRIMARY 11 [failed] 1 0 10 1
- A target group to connect the load balancer to your service [delete complete] [0.0s]
- An ECS task definition to group your containers and run them on ECS [delete complete] [1.2s]
- An IAM role to control permissions for the containers in your tasks [delete complete] [2.4s]
✘ execute svc deploy: deploy service front-end to environment production: deploy service: stack webapp-production-front-end did not complete successfully and exited with status ROLLBACK_COMPLETE
Dockerfile
# prod.Dockerfile
FROM ruby:3.0
# curl commands for installing yarn (https://stackoverflow.com/a/59414701/3858363)
RUN curl https://deb.nodesource.com/setup_12.x | bash
RUN curl https://dl.yarnpkg.com/debian/pubkey.gpg | apt-key add -
RUN echo "deb https://dl.yarnpkg.com/debian/ stable main" | tee /etc/apt/sources.list.d/yarn.list
RUN apt-get update -qq && apt-get install -y nodejs postgresql-client yarn
WORKDIR /webapp
COPY . /webapp
RUN bundle install
# Add a script to be executed every time the container starts.
COPY entrypoint.sh /usr/bin/
RUN chmod +x /usr/bin/entrypoint.sh
ENTRYPOINT ["entrypoint.sh"]
EXPOSE 3000
RUN ./bin/bundle add jsbundling-rails
RUN ./bin/rails javascript:install:esbuild
RUN ./bin/rails css:install:sass
# environment specific commands
ENV RAILS_ENV production
ENV RAILS_SERVE_STATIC_FILES true
ENV RAILS_LOG_TO_STDOUT true
RUN rake assets:precompile
CMD ["rails", "server", "-e", "production", "-b", "0.0.0.0"]
manifest.yml
name: front-end
type: Load Balanced Web Service
# Distribute traffic to your service.
http:
# Requests to this path will be forwarded to your service.
# To match all requests you can use the "/" path.
path: '/'
# You can specify a custom health check path. The default is "/".
# healthcheck: '/'
# Configuration for your containers and service.
image:
# Docker build arguments. For additional overrides: https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/#image-build
build: prod.Dockerfile
# Port exposed through your container to route traffic to it.
port: 3000
cpu: 256 # Number of CPU units for the task.
memory: 512 # Amount of memory in MiB used by the task.
platform: linux/arm64 # See https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/#platform
count: 1 # Number of tasks that should be running in your service.
exec: true # Enable running commands in your container.
Hello, @pgrennin!
Let's see if we can find what happened. Can you go back into the ECS console, click on your cluster, then click on the 'Tasks' tab, and choose Stopped for "Desired task status:"?
What do you see there?
Hello @huanjani, I just took a look at my stopped tasks but it's not showing any tasks. I would expect the tasks to show up here. Maybe they get deleted as part of the rollback?
So I found the applicable deploy logs in CloudWatch > Log Groups > /copilot/my-app-front-end
It says the error is "A server is already running. Check /webapp/tmp/pids/server.pid."
So I'm going to try removing the pid in my deploy script and will retry deploy.
Hello @pgrennin. Just a reminder if you don't want your resources to get automatically deleted because of CFN rollback, you could specify --no-rollback flag when doing svc deploy.
I just took a look at my stopped tasks but it's not showing any tasks.
This might be because it's been a while since the last task stopped and they got cleaned by ECS automatically (see here)
@huanjani, Yes, I was able to deploy after fixing a few issues. I just needed to find where the deploy logs were kept so I could see the deploy error. Consider adding an error message that logs can be viewed in CloudWatch if there is an error during deployment or even better add the error message in the CLI. I think AWS Copilot is a cool product.
ECS
Thank you @huanjani
I was able to find those in the ECS -> Clusters -> Tasks -> tasks details
Stopped
Stopped reason
ResourceInitializationError: unable to pull secrets or registry auth: execution resource retrieval failed: unable to retrieve secrets from ssm: service call has been retried 1 time(s): AccessDeniedException: User: arn:aws:sts::783882260760:assumed-role/todo-dev-api-ExecutionRole-1DEF962Y4MDN2/0fa0ee6a54b64e449b1837216c1bfc60 is not authorized to perform: ssm:GetParameters on resource: arn:aws:ssm:us-east-2:783882260760:parameter/copilot/todo/dev/secrets/MONGODB_URI because no identity-based policy allows the ssm:GetParameters action status code: 400, request id: 38d04c7c-0694-47fb-a95b-6f9e7c5c49d3
This issue is stale because it has been open 60 days with no response activity. Remove the stale label, add a comment, or this will be closed in 14 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}