copilot-cli icon indicating copy to clipboard operation
copilot-cli copied to clipboard

Pipeline fails in CodeBuild POST_BUILD with `COMMAND_EXECUTION_ERROR Message: Error while executing command: for env in $pl_envs;`

Open ryancody opened this issue 2 years ago • 11 comments

Hello, I'm using Copilot in windows for a dotnet project. I have a pipeline that that fails in CodeBuild on the POST_BUILD phase with the following error:

image

Build logs shows this error: [Container] 2022/05/11 04:20:06 Phase context status code: COMMAND_EXECUTION_ERROR Message: Error while executing command: for env in $pl_envs; do tag=$(sed 's/:/-/g' <<<"${CODEBUILD_BUILD_ID##*:}-${env}" | rev | cut -c 1-128 | rev) for svc in $svcs; do ./copilot-linux svc package -n $svc -e $env --output-dir './infrastructure' --tag $tag --upload-assets; if [ $? -ne 0 ]; then echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2; exit 1; fi done; for job in $jobs; do ./copilot-linux job package -n $job -e $env --output-dir './infrastructure' --tag $tag --upload-assets; if [ $? -ne 0 ]; then echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2; exit 1; fi done; done; . Reason: exit status 1

It refers to this line in buildspec.yml: image

ryancody avatar May 11 '22 04:05 ryancody

Hi, @ryancody! Would you mind copy-pasting your manifest here? Thanks!

huanjani avatar May 11 '22 15:05 huanjani

# The manifest for the "projectname-master" pipeline.
# This YAML file defines your pipeline: the source repository it tracks and the order of the environments to deploy to.
# For more info: https://aws.github.io/copilot-cli/docs/manifest/pipeline/

# The name of the pipeline.
name: projectname-master

# The version of the schema used in this template.
version: 1

# This section defines your source, changes to which trigger your pipeline.
source:
  # The name of the provider that is used to store the source artifacts.
  # (i.e. GitHub, Bitbucket, CodeCommit)
  provider: GitHub
  # Additional properties that further specify the location of the artifacts.
  properties:
    branch: master
    repository: https://github.com/ryancody/projectname
    # Optional: specify the name of an existing CodeStar Connections connection.
    # connection_name: a-connection

# This section defines the order of the environments your pipeline will deploy to.
stages:
    - # The name of the environment.
      name: test
      # Optional: flag for manual approval action before deployment.
      # requires_approval: true
      # Optional: use test commands to validate this stage of your build.
      # test_commands: [echo 'running tests', make test]

ryancody avatar May 11 '22 18:05 ryancody

Thanks! Can you also paste here the manifests for the workloads (services/jobs) that you're attempting to deploy with this pipeline?

huanjani avatar May 11 '22 18:05 huanjani

service 1

# The manifest for the "projectname" service.
# Read the full specification for the "Backend Service" type at:
#  https://aws.github.io/copilot-cli/docs/manifest/backend-service/

# Your service name will be used in naming your resources like log groups, ECS services, etc.
name: projectname
type: Backend Service

# Your service is reachable at "http://projectname.${COPILOT_SERVICE_DISCOVERY_ENDPOINT}:5024" but is not public.

# Configuration for your containers and service.
image:
  # Docker build arguments. For additional overrides: https://aws.github.io/copilot-cli/docs/manifest/backend-service/#image-build
  build: src\Dockerfile
  # Port exposed through your container to route traffic to it.
  port: 5024

cpu: 256       # Number of CPU units for the task.
memory: 512    # Amount of memory in MiB used by the task.
count: 1       # Number of tasks that should be running in your service.
exec: true     # Enable running commands in your container.

# Optional fields for more advanced use-cases.
#
#variables:                    # Pass environment variables as key value pairs.
#  LOG_LEVEL: info

#secrets:                      # Pass secrets from AWS Systems Manager (SSM) Parameter Store.
#  GITHUB_TOKEN: GITHUB_TOKEN  # The key is the name of the environment variable, the value is the name of the SSM parameter.

# You can override any of the values defined above by environment.
#environments:
#  test:
#    count: 2               # Number of tasks to run for the "test" environment.
#    deployment:            # The deployment strategy for the "test" environment.
#       rolling: 'recreate' # Stops existing tasks before new ones are started for faster deployments.

service 2

# The manifest for the "api" service.
# Read the full specification for the "Request-Driven Web Service" type at:
# https://aws.github.io/copilot-cli/docs/manifest/rd-web-service/

# Your service name will be used in naming your resources like log groups, App Runner services, etc.
name: api
# The "architecture" of the service you're running.
type: Request-Driven Web Service

image:
  # Docker build arguments.
  # For additional overrides: https://aws.github.io/copilot-cli/docs/manifest/rd-web-service/#image-build
  build: src\Dockerfile
  # Port exposed through your container to route traffic to it.
  port: 5023

# http:
#   healthcheck:
#     path: /
#     healthy_threshold: 3
#     unhealthy_threshold: 5
#     interval: 10s
#     timeout: 5s

# Number of CPU units for the task.
cpu: 1024
# Amount of memory in MiB used by the task.
memory: 2048

# # Connect your App Runner service to your environment's VPC.
# network:
#   vpc:
#     placement: private

# Enable tracing for the service.
# observability:
#   tracing: awsxray

# Optional fields for more advanced use-cases.
#
# variables:                    # Pass environment variables as key value pairs.
#   LOG_LEVEL: info
#
# tags:                         # Pass tags as key value pairs.
#   project: project-name

# You can override any of the values defined above by environment.
# environments:
#   test:
#     variables:
#       LOG_LEVEL: debug        # Log level for the "test" environment.

ryancody avatar May 11 '22 20:05 ryancody

Thanks! Some troubleshooting questions:

  • You're on Windows; is the image you're building a Windows image? If so, you'll need platform: windows/x86_64 in your workload manifest. Also, unfortunately, App Runner (Request-Driven Web Services) doesn't support Windows at this time.
  • Have you pushed all of your files up to your repo?
  • Have you deployed the test environment?
  • Have you been able to deploy these services to the environment manually, not through the pipeline?
  • Would it work if the Dockerfile were not within the src dir, but at the level you're deploying from? I'm curious about if the forward slash is the problem-- if it does indeed work if you move the Dockerfile, I will dig into this problem and figure out the solution so that you can have it within the src dir in the future.

huanjani avatar May 11 '22 21:05 huanjani

@huanjani sorry - i was down for a few days with a cold, i'll get back to you on this once i get caught up and compile this info

ryancody avatar May 16 '22 13:05 ryancody

No rush! Hope you're feeling better!

huanjani avatar May 16 '22 20:05 huanjani

  • I'm using mcr.microsoft.com/dotnet/aspnet images, I believe they are either Alpine, Debian or Ubuntu
  • yes, all files are commited and pushed to my repo
  • yes, copilot env show shows my env deployed in us-east-1
  • yes, I can deploy them manually but not through the pipeline
  • I believe i've tried moving the dockerfile to the root and back but that might have been before the v1.18 fix. I'll try this again.

ryancody avatar May 19 '22 02:05 ryancody

I tried launching from scratch from WSL with source code and Dockerfile in the root folder, still getting the same error.

ryancody avatar May 19 '22 05:05 ryancody

The CodeBuild environment is a linux environment. That's probably why you were able to deploy it manually (with local dev env being Windows), but not able to deploy it through pipeline. @huanjani's suggestion to add a platform: windows/x86_64 may work for you! For more information on this field you can check out here!

Lou1415926 avatar May 23 '22 17:05 Lou1415926

I am experiencing a similar error when I use the Ubuntu aws/codebuild/standard:6.0 image in CodeBuild. Here is an extract from my logs:

[Container] 2022/08/20 00:11:31 Running command for env in $pl_envs; do
  tag=$(sed 's/:/-/g' <<<"${CODEBUILD_BUILD_ID##*:}-${env}" | rev | cut -c 1-128 | rev)
  for svc in $svcs; do
  ./copilot-linux svc package -n $svc -e $env --output-dir './infrastructure' --tag $tag --upload-assets;
  if [ $? -ne 0 ]; then
    echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
    exit 1;
  fi
  done;
  for job in $jobs; do
  ./copilot-linux job package -n $job -e $env --output-dir './infrastructure' --tag $tag --upload-assets;
  if [ $? -ne 0 ]; then
    echo "Cloudformation stack and config files were not generated. Please check build logs to see if there was a manifest validation error." 1>&2;
    exit 1;
  fi
  done;
done;
/codebuild/output/tmp/script.sh: 5: Syntax error: redirection unexpected
[Container] 2022/08/20 00:11:31 Command did not exit successfully for env in $pl_envs; do

I could not 100% confirm this but my hypothesis is that the commands are running in sh instead of bash and thus the <<< in the second line does not parse.

Since the generated buildspec is already using Ruby in other parts of the build I replaced it with a more portable version and it works correctly for me now.

tag=$(ruby -e 'puts ARGV[0].split(":").last[0..128]' "${CODEBUILD_BUILD_ID}-${env}")

federicobond avatar Aug 20 '22 01:08 federicobond

This issue is stale because it has been open 60 days with no response activity, and is tagged with pending/question. Remove the stale label, add a comment, or this will be closed in 14 days.

github-actions[bot] avatar Oct 20 '22 00:10 github-actions[bot]

For those who facing issues in setting pipeline for Copilot in AWS we have moved the pipeline to Bitbucket pipelines for time being. Attaching the link to the discussion for the same. https://github.com/aws/copilot-cli/discussions/4124#discussioncomment-4025370

iamsajithk avatar Nov 01 '22 08:11 iamsajithk

This issue is stale because it has been open 60 days with no response activity, and is tagged with pending/question. Remove the stale label, add a comment, or this will be closed in 14 days.

github-actions[bot] avatar Jan 02 '23 00:01 github-actions[bot]

This issue is closed due to inactivity. Feel free to reopen the issue if you have any follow-ups!

github-actions[bot] avatar Jan 16 '23 00:01 github-actions[bot]

The issue arises when attempting to loop through $pl_envs while having only one environment created. To address this, consider either creating additional environments or adjusting the YAML file to function seamlessly with a single environment.

Taronm98 avatar Nov 23 '23 13:11 Taronm98

same problem here and I have 3 environments:

$copilot env ls
dev
prod
staging

each env is on a different account. The app is on the prod account.

sebastianovide avatar Dec 05 '23 22:12 sebastianovide