Builds suddenly failing with "Executing custom build scripts is not yet implemented"

[andyfries]

Before opening, please confirm:

• [X] I have checked to see if my question is addressed in the FAQ.
• [X] I have searched for duplicate or closed issues.
• [X] I have read the guide for submitting bug reports.
• [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• [X] I have removed any sensitive information from my code snippets and submission.

App Id

d30alijgcsdzu9

AWS Region

us-east-1

Amplify Hosting feature

Backend builds, Build settings

Frontend framework

Next.js

Next.js version

13.4.19

Next.js router

Pages Router

Describe the bug

I'm experiencing the same issue described in https://github.com/aws-amplify/amplify-hosting/issues/3670 where my backend build phase has begun failing after upgrading my Amplify CLI from 11.x to v12.8.2 using amplify upgrade. The exact same configuration had been working fine previously and local amplify push commands continue to work.

I've attempted both workarounds suggested by an Amplify team member in https://github.com/aws-amplify/amplify-hosting/issues/3670#issuecomment-1714620505 as well as various other attempts at downgrading my local CLI version (which then failed with the same error) and using live package updates to override the Node version (which failed with all sorts of odd NPM behavior).

The most success I had was using CLI v12.8.2 in the build image which seemingly worked all the way up until partway through the test phase when it timed out trying to contact the local server endpoint (visible in build 84 of my project). However, closer inspection shows that the frontend build step inexplicably skipped several important steps which likely caused the inability to start the server (compare with build 44, the last successful build).

I've submitted the results from amplify diagnose under Project Identifier eb3c0191d4cfe11e7ee1aa7996271c9d.

Expected behavior

Builds were working with the exact same settings before updating my Amplify CLI version from v11.x to v12.8.2

Reproduction steps

I'm not sure of the exact steps to reproduce because I'm not sure how my project got into this state in the first place. I would expect you could get into the same state by using CLI v11.x to create and push a project, then upgrade to the latest version (v12.8.2 at time of writing) and push again.

Build Settings

No live package overrides (besides those added temporarily to try to work around the issue).

amplify.yml

version: 1
backend:
  phases:
    preBuild:
      commands:
        - export CGO_ENABLED=0
    build:
      commands:
        - '# Execute Amplify CLI with the helper script'
        - amplifyPush --simple
frontend:
  phases:
    preBuild:
      commands:
        - npm ci
    build:
      commands:
        - export NEXT_PUBLIC_SENTRY_RELEASE=${AWS_COMMIT_ID}
        - export NEXT_PUBLIC_SENTRY_ENVIRONMENT=$SENTRY_ENVIRONMENT
        - npm run build
  artifacts:
    baseDirectory: .next
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*
      - .next/cache/**/*
test:
  phases:
    preTest:
      commands:
        - npm ci
        - npm install mocha mochawesome mochawesome-merge mochawesome-report-generator
        - pm2 start npm -- start
        - wait-on http://localhost:3000
    test:
      commands:
        - 'npx cypress run --reporter mochawesome --reporter-options "reportDir=cypress/report/mochawesome-report,overwrite=false,html=false,json=true,timestamp=mmddyyyy_HHMMss"'
    postTest:
      commands:
        - npx mochawesome-merge cypress/report/mochawesome-report/mochawesome*.json > cypress/report/mochawesome.json
        - pm2 kill
  artifacts:
    baseDirectory: cypress
    configFilePath: '**/mochawesome.json'
    files:
      - '**/*.png'
      - '**/*.mp4'

Custom build image Dockerfile

I'm using a custom build image because my project contains some Go code. The Dockerfile for that is included below.

FROM amazonlinux:2023

# Most of the structure and commands taken from
# https://github.com/aws-amplify/amplify-hosting/blob/6f1b089c35b242defee5661c83e5c161249c46b5/images/latest/Dockerfile
# later versions of packages were used where it made sense, and unsed packages were removed

# install basic tools, and to also keep a similar structure to amazon amplify build
RUN touch ~/.bashrc
RUN yum -y update && \
    yum -y install \
      alsa-lib \
      autoconf \
      automake \
      git \
      gtk3-devel \
      golang \
      libnotify-devel \
      libtool \
      libXScrnSaver \
      nss \
      tar \
      wget \
      xorg-x11-server-Xvfb && \
    rm -rf /var/cache/yum

# install nvm
RUN curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash

# install nodejs 18.16.0 is the LTS version
RUN /bin/bash -c ". ~/.nvm/nvm.sh && nvm install 18.16.0"

WORKDIR /usr/app
RUN echo "lts/*" > .nvmrc

# install cypress for testing, and other tools used for running tests
# everything after cypress in this layer is used for running tests, and not part of the original amplify build
RUN /bin/bash -c ". ~/.nvm/nvm.sh && \
    npm install -g --unsafe-perm=true --allow-root [email protected] && \
    npm install -g pm2 && \
    npm install -g wait-on"

# install amplify cli
RUN /bin/bash -c ". ~/.nvm/nvm.sh && \
    npm install -g @aws-amplify/[email protected]"

# make sure nvm is in $PATH and up to date .bashrc
RUN echo export PATH="\
/root/.nvm/versions/node/18.16.0/bin:\
$PATH" >> ~/.bashrc && \
    echo "nvm use 18.16.0 1> /dev/null" >> ~/.bashrc

ENTRYPOINT [ "bash", "-c" ]

Log output

Failure during Backend build phase

2023-12-12T23:03:38.706Z [INFO]: 🛑 Executing custom build scripts is not yet implemented
2023-12-12T23:03:38.706Z [INFO]: Learn more at: https://docs.amplify.aws/cli/project/troubleshooting/
2023-12-12T23:03:38.798Z [WARNING]: - Loading layer data from the cloud...
2023-12-12T23:03:38.840Z [INFO]: 
2023-12-12T23:03:38.840Z [INFO]: Session Identifier: 26ff9ab7-1c9e-43fe-aa3e-95115067d900
2023-12-12T23:03:38.871Z [ERROR]: !!! Build failed
2023-12-12T23:03:38.872Z [INFO]: Please read more about Amplify Hosting's support for SSR frameworks to find if your build failure is related to an unsupported feature: https://docs.aws.amazon.com/amplify/latest/userguide/amplify-ssr-framework-support.html. You may also find this troubleshooting guide useful: https://docs.aws.amazon.com/amplify/latest/userguide/troubleshooting-ssr-deployment.html
2023-12-12T23:03:38.872Z [ERROR]: !!! Non-Zero Exit Code detected
2023-12-12T23:03:38.872Z [INFO]: # Starting environment caching...
2023-12-12T23:03:38.872Z [INFO]: # Uploading environment cache artifact...
2023-12-12T23:03:38.977Z [INFO]: # Uploaded environment cache artifact
2023-12-12T23:03:38.977Z [INFO]: # Environment caching completed
Terminating logging...

Additional information

This appears to be the same issue as https://github.com/aws-amplify/amplify-hosting/issues/3670. I also submitted a support case through the AWS console as Case ID 170266866701174.

[e-simpson]

Seems to me that your Amplify build isn't able to correctly use your custom image.

To cover some bases here:

• Did your custom docker image build correctly?
• Have you set its public repository reference in your Amplify app's build image settings? Note: you can do this via the _CUSTOM_IMAGE Amplify environment variable as well.
• Does Amplify have access to the built image repo? It has to either be a publicly accessible repo or you'll have to workout the permissions so Amplify can pull it.

[mauerbac]

Closing since we havent heard back. Please re-open a ticket with AWS support if you are having an issue

[github-actions[bot]]

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.

[github-actions[bot]]

This issue has been automatically locked.