amplify-backend icon indicating copy to clipboard operation
amplify-backend copied to clipboard

Deployment Pipeline failing randomly TimeoutError: {"state":"TIMEOUT","reason":"Waiter has timed out"}

Open renatofrias26 opened this issue 1 year ago • 8 comments

Environment information

System:
  OS: macOS 14.5
  CPU: (8) arm64 Apple M1 Pro
  Memory: 205.28 MB / 16.00 GB
  Shell: /bin/zsh
Binaries:
  Node: 18.20.1 - /usr/local/bin/node
  Yarn: 1.22.19 - /usr/local/bin/yarn
  npm: 10.8.2 - /usr/local/bin/npm
  pnpm: undefined - undefined
NPM Packages:
  @aws-amplify/backend: 1.0.4
  @aws-amplify/backend-cli: 1.2.0
  aws-amplify: 6.4.0
  aws-cdk: 2.148.0
  aws-cdk-lib: 2.148.0
  typescript: 5.4.5
AWS environment variables:
  AWS_STS_REGIONAL_ENDPOINTS = regional
  AWS_NODEJS_CONNECTION_REUSE_ENABLED = 1
  AWS_SDK_LOAD_CONFIG = 1
No CDK environment variables

Data packages

├─┬ @aws-amplify/[email protected]
│ └─┬ @aws-amplify/[email protected]
│   └── @aws-amplify/[email protected]
└─┬ @aws-amplify/[email protected]
  └─┬ @aws-amplify/[email protected]
    └── @aws-amplify/[email protected]

Description

My pipeline is failing multiple times without much explanation, re-triggering the pipeline seems to solve the issue eventually. The issue stated happening after adding Google & Apple social sign in.

Error:

amplify-xxxx-xxxx-xxxx | 9:11:07 AM | [31mUPDATE_FAILED [39m | Custom::SecretFetcherResource | [31m[1mauth/SIWA_CLIENT_IDSecretFetcherResource/Default[22m[39m (SIWACLIENTIDSecretFetcherResource) [31m[1mReceived response status [FAILED] from custom resource. Message returned: TimeoutError: {"state":"TIMEOUT","reason":"Waiter has timed out"}

Auth resource.ts:

import { defineAuth, secret } from "@aws-amplify/backend";
import { preSignUp } from "./link-user-provider/resource";

export const auth = defineAuth({
  loginWith: {
    email: {
      verificationEmailStyle: "CODE",
      verificationEmailSubject: "Welcome to Autobook!",
      verificationEmailBody: (createCode) =>
        `Welcome to Autobook! Use this code to confirm your account: ${createCode()}`,
    },
    externalProviders: {
      google: {
        clientId: secret("GOOGLE_CLIENT_ID"),
        clientSecret: secret("GOOGLE_CLIENT_SECRET"),
        scopes: ["email"],
      },
      signInWithApple: {
        clientId: secret("SIWA_CLIENT_ID"),
        keyId: secret("SIWA_KEY_ID"),
        privateKey: secret("SIWA_PRIVATE_KEY"),
        teamId: secret("SIWA_TEAM_ID"),
      },
      callbackUrls: [
        "http://localhost:8100/login",
        "https://int-app.autobook.au/login",
        "https://uat-app.autobook.au/login",
      ],
      logoutUrls: [
        "http://localhost:8100/login",
        "https://int-app.autobook.au/login",
        "https://uat-app.autobook.au/login",
      ],
    },
  },
  triggers: {
    preSignUp,
  },
  userAttributes: {},
});

data resource.ts:

import { type ClientSchema, a, defineData } from '@aws-amplify/backend';
import { updateUserCognitoEmail } from '../functions/updateUserCognitoEmail/resource';

const schema = a
  .schema({
    Vehicle: a
      .model({
        vehicleId: a.id().required(),
        generalDescription: a.string(),
        registration: a.customType({
          state: a.string(),
          number: a.string(),
        }),
        vin: a.string(),
        color: a.string(),
        bodyType: a.string(),
        nvic: a.string(),
        make: a.string(),
        model: a.string(),
        year: a.integer(),
        variant: a.string(),
        series: a.string(),
        summary: a.string(),
        features: a.string().array(),
        transmission: a.string(),
        engineType: a.string(),
        engineCC: a.string(),
        engineLitre: a.string(),
        fuelType: a.string(),
        cylinder: a.string(),
        horsePower: a.string(),
        torque: a.string(),
        price: a.customType({
          privateSale: a.customType({
            from: a.integer(),
            to: a.integer(),
          }),
          dealerRetail: a.customType({
            from: a.integer(),
            to: a.integer(),
          }),
          dealerTrade: a.customType({
            from: a.integer(),
            to: a.integer(),
          }),
        }),
        mods: a.string().array(),
        odometer: a.integer(),
        name: a.string(),
        userId: a.string().required(),
        user: a.belongsTo('User', 'userId'),
        profileImg: a.string(),
        heroImg: a.string(),
        gallery: a.hasOne('VehicleGallery', 'vehicleId'),
        followedByUsers: a.hasMany('UserVehicleFollow', 'vehicleId'),
      })
      .secondaryIndexes((index) => [index('vehicleId')])
      .authorization((allow) => [
        allow.owner(),
        allow.guest().to(['read']),
        allow.authenticated().to(['read']),
        allow.publicApiKey().to(['read']),
      ]),

    User: a
      .model({
        id: a.id().required(),
        email: a.email(),
        username: a.string().required(),
        name: a.string(),
        interests: a.string().array(),
        profileType: a.enum([
          'CAR_OWNER',
          'CAR_ENTHUSIAST',
          'WORKSHOPS',
          'DEALERS',
          'SHOPS',
        ]),
        location: a.customType({
          state: a.string(),
          postcode: a.integer(),
          suburb: a.string(),
        }),
        instagram: a.string(),
        facebook: a.string(),
        currentVehicles: a.hasMany('Vehicle', 'userId'),
        bio: a.string(),
        profileImg: a.string(),
        heroImg: a.string(),
        vehicleComments: a.hasMany('VehicleComment', 'userId'),
        vehicleReactions: a.hasMany('VehicleReaction', 'userId'),
        userComments: a.hasMany('UserComment', 'userId'),
        userReactions: a.hasMany('UserReaction', 'userId'),
        gallery: a.hasOne('UserGallery', 'userId'),
        isProfileComplete: a.boolean(),
        vehicleFollows: a.hasMany('UserVehicleFollow', 'userId'),
        notifications: a.hasMany('UserNotification', 'userId'),
      })
      .secondaryIndexes((index) => [index('email'), index('username')])
      .authorization((allow) => [
        allow.owner(),
        allow.guest().to(['read']),
        allow.authenticated().to(['read']),
        allow.publicApiKey().to(['read']),
      ]),

    UserVehicleFollow: a
      .model({
        id: a.id().required(),
        userId: a.string().required(),
        user: a.belongsTo('User', 'userId'),
        vehicleId: a.string().required(),
        vehicle: a.belongsTo('Vehicle', 'vehicleId'),
      })
      .secondaryIndexes((index) => [index('userId'), index('vehicleId')]),

    VehicleGallery: a
      .model({
        id: a.id().required(),
        title: a.string().required(),
        vehicleId: a.string().required(),
        vehicle: a.belongsTo('Vehicle', 'vehicleId'),
        galleryImages: a.hasMany('VehicleGalleryImage', 'galleryId'),
      })
      .secondaryIndexes((index) => [index('vehicleId')]),

    VehicleGalleryImage: a.model({
      id: a.id().required(),
      url: a.string().required(),
      comments: a.hasMany('VehicleComment', 'galleryImageId'),
      reactions: a.hasMany('VehicleReaction', 'galleryImageId'),
      galleryId: a.string().required(),
      gallery: a.belongsTo('VehicleGallery', 'galleryId'),
    }),

    VehicleComment: a
      .model({
        id: a.id().required(),
        parentId: a.string(),
        parent: a.belongsTo('VehicleComment', 'parentId'),
        content: a.string().required(),
        userId: a.string().required(),
        user: a.belongsTo('User', 'userId'),
        galleryImageId: a.string(),
        galleryImage: a.belongsTo('VehicleGalleryImage', 'galleryImageId'),
        replyGalleryImageId: a.string(),
        replies: a.hasMany('VehicleComment', 'parentId'),
      })
      .secondaryIndexes((index) => [index('galleryImageId')]),

    VehicleReaction: a.model({
      id: a.id().required(),
      type: a.enum(['like', 'love', 'laugh', 'wow', 'sad', 'angry']),
      userId: a.string().required(),
      user: a.belongsTo('User', 'userId'),
      galleryImageId: a.string().required(),
      galleryImage: a.belongsTo('VehicleGalleryImage', 'galleryImageId'),
    }),

    UserGallery: a
      .model({
        id: a.id().required(),
        title: a.string().required(),
        userId: a.string().required(),
        user: a.belongsTo('User', 'userId'),
        galleryImages: a.hasMany('UserGalleryImage', 'galleryId'),
      })
      .secondaryIndexes((index) => [index('userId')]),

    UserGalleryImage: a.model({
      id: a.id().required(),
      url: a.string().required(),
      comments: a.hasMany('UserComment', 'galleryImageId'),
      reactions: a.hasMany('UserReaction', 'galleryImageId'),
      galleryId: a.string().required(),
      gallery: a.belongsTo('UserGallery', 'galleryId'),
    }),

    UserComment: a
      .model({
        id: a.id().required(),
        parentId: a.string(),
        parent: a.belongsTo('UserComment', 'parentId'),
        content: a.string().required(),
        userId: a.string().required(),
        user: a.belongsTo('User', 'userId'),
        galleryImageId: a.string(),
        galleryImage: a.belongsTo('UserGalleryImage', 'galleryImageId'),
        replyGalleryImageId: a.string(),
        replies: a.hasMany('UserComment', 'parentId'),
      })
      .secondaryIndexes((index) => [index('galleryImageId')]),

    UserReaction: a.model({
      id: a.id().required(),
      type: a.enum(['like', 'love', 'laugh', 'wow', 'sad', 'angry']),
      userId: a.string().required(),
      user: a.belongsTo('User', 'userId'),
      galleryImageId: a.string().required(),
      galleryImage: a.belongsTo('UserGalleryImage', 'galleryImageId'),
    }),

    UserNotification: a
      .model({
        id: a.id().required(),
        userId: a.string().required(),
        user: a.belongsTo('User', 'userId'),
        vehicleId: a.string(),
        message: a.string().required(),
        type: a.enum(['comment', 'reaction', 'follow']),
        url: a.string(),
        photoUrl: a.string(),
        isRead: a.boolean().default(false),
      })
      .secondaryIndexes((index) => [index('userId')])
      .authorization((allow) => [allow.owner(), allow.authenticated()]),

    updateUserCognitoEmail: a
      .query()
      .arguments({
        email: a.email(),
      })
      .returns(a.customType({ statusCode: a.integer(), body: a.string() }))
      .handler(a.handler.function(updateUserCognitoEmail))
      .authorization((allow) => [allow.authenticated(), allow.publicApiKey()]),
  })
  .authorization((allow) => [
    allow.owner(),
    allow.guest().to(['read']),
    allow.authenticated().to(['read']),
    allow.publicApiKey().to(['read']),
    allow.resource(updateUserCognitoEmail),
  ]);

export type Schema = ClientSchema<typeof schema>;

export const data = defineData({
  schema,
  authorizationModes: {
    defaultAuthorizationMode: 'userPool',
    // API Key is used for a.allow.public() rules
    apiKeyAuthorizationMode: {
      expiresInDays: 30,
    },
  },
});

Let me know if you need any extra details, it seems that if remove social providers, builds fine every time.

renatofrias26 avatar Jul 31 '24 09:07 renatofrias26

Hi @renatofrias26, are you still experiencing this issue? Yesterday, there was a large disruption to many AWS services that Amplify relies on. This could have caused the deploy failures that you are describing.

dpilch avatar Jul 31 '24 15:07 dpilch

Hi @dpilch, I don't think it's related. I've been having this issue for over 10 days. I have created this application when Gen 2 was still beta, but I did update as recommended. I wondering if there is still something missing in the configs that was added later...

image

renatofrias26 avatar Jul 31 '24 21:07 renatofrias26

Ok, we will try to reproduce this issue and report back.

dpilch avatar Jul 31 '24 21:07 dpilch

@dpilch, FYI I have just disabled Apple sign in and pipeline seems happy again.

renatofrias26 avatar Jul 31 '24 21:07 renatofrias26

Hey @renatofrias26, We are working on reproducing the issue. Is it only affecting Apple sign-in?

AnilMaktala avatar Aug 02 '24 20:08 AnilMaktala

Hi @AnilMaktala, yes. As soon as I deactivate Apple sign in the pipeline runs no problems. Happy to share any extra details you may need!

renatofrias26 avatar Aug 02 '24 20:08 renatofrias26

Hey @renatofrias26 ,👋 thanks for confirming! I'm going to transfer this over to our Backend repository for better assistance 🙂.

AnilMaktala avatar Aug 06 '24 14:08 AnilMaktala

Hey @renatofrias26, the issue appears to be similar to https://github.com/aws-amplify/amplify-backend/issues/1797. Refer to the comment on https://github.com/aws-amplify/amplify-backend/issues/1797#issuecomment-2260534958 Check the concurrent execution -> Applied account-level quota value on the Service quota console. you may need to request a quota increase to mitigate the issue as the account quota maybe set to 10 concurrency similar to the comment

ykethan avatar Aug 06 '24 14:08 ykethan

Closing the issue due to inactivity. Do reach out to us if you are still experiencing this issue.

ykethan avatar Aug 21 '24 14:08 ykethan