Gen2: DynamoDB Streams - CloudFormation fails due to Circular dependency between resources

[jgo80]

Before opening, please confirm:

• [X] I have searched for duplicate or closed issues and discussions.
• [X] I have read the guide for submitting bug reports.
• [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.

JavaScript Framework

Not applicable

Amplify APIs

GraphQL API

Amplify Version

v6

Amplify Categories

auth, function, api

Backend

Amplify Gen 2 (Preview)

Environment information

# Put output below this line
  System:
    OS: macOS 14.4.1
    CPU: (10) arm64 Apple M1 Max
    Memory: 4.63 GB / 32.00 GB
    Shell: 5.9 - /bin/zsh
  Binaries:
    Node: 22.1.0 - /opt/homebrew/bin/node
    Yarn: 1.22.22 - /opt/homebrew/bin/yarn
    npm: 10.7.0 - /opt/homebrew/bin/npm
    Watchman: 2024.05.06.00 - /opt/homebrew/bin/watchman
  Browsers:
    Chrome: 124.0.6367.119
    Safari: 17.4.1
  npmPackages:
    %name%:  0.1.0 
    @aws-amplify/backend: ^1.0.0 => 1.0.1 
    @aws-amplify/backend-cli: ^1.0.1 => 1.0.2 
    @aws-amplify/react-native: ^1.1.0 => 1.1.0 
    @aws-amplify/ui-react-native: ^2.1.6 => 2.2.0 
    @aws-lambda-powertools/logger: ^2.1.0 => 2.1.0 
    @babel/core: ^7.20.0 => 7.24.5 
    @expo/vector-icons: ^14.0.0 => 14.0.1 
    @react-native-async-storage/async-storage: ^1.23.1 => 1.23.1 
    @react-native-community/netinfo: ^11.3.1 => 11.3.1 
    @react-navigation/native: ^6.0.2 => 6.1.17 
    @types/aws-lambda: ^8.10.137 => 8.10.137 
    @types/jest: ^29.5.12 => 29.5.12 
    @types/react: ~18.2.45 => 18.2.79 (18.3.1)
    @types/react-test-renderer: ^18.0.7 => 18.3.0 
    HelloWorld:  0.0.1 
    aws-amplify: ^6.3.0 => 6.3.0 
    aws-amplify/adapter-core:  undefined ()
    aws-amplify/analytics:  undefined ()
    aws-amplify/analytics/kinesis:  undefined ()
    aws-amplify/analytics/kinesis-firehose:  undefined ()
    aws-amplify/analytics/personalize:  undefined ()
    aws-amplify/analytics/pinpoint:  undefined ()
    aws-amplify/api:  undefined ()
    aws-amplify/api/server:  undefined ()
    aws-amplify/auth:  undefined ()
    aws-amplify/auth/cognito:  undefined ()
    aws-amplify/auth/cognito/server:  undefined ()
    aws-amplify/auth/enable-oauth-listener:  undefined ()
    aws-amplify/auth/server:  undefined ()
    aws-amplify/data:  undefined ()
    aws-amplify/data/server:  undefined ()
    aws-amplify/datastore:  undefined ()
    aws-amplify/in-app-messaging:  undefined ()
    aws-amplify/in-app-messaging/pinpoint:  undefined ()
    aws-amplify/push-notifications:  undefined ()
    aws-amplify/push-notifications/pinpoint:  undefined ()
    aws-amplify/storage:  undefined ()
    aws-amplify/storage/s3:  undefined ()
    aws-amplify/storage/s3/server:  undefined ()
    aws-amplify/storage/server:  undefined ()
    aws-amplify/utils:  undefined ()
    aws-cdk: ^2.140.0 => 2.141.0 
    aws-cdk-lib: ^2.140.0 => 2.141.0 
    constructs: ^10.3.0 => 10.3.0 
    esbuild: ^0.21.1 => 0.21.1 (0.20.2)
    expo: ~51.0.0 => 51.0.2 
    expo-constants: ~16.0.1 => 16.0.1 
    expo-font: ~12.0.4 => 12.0.4 
    expo-linking: ~6.3.1 => 6.3.1 
    expo-router: ~3.5.10 => 3.5.11 
    expo-splash-screen: ~0.27.4 => 0.27.4 
    expo-status-bar: ~1.12.1 => 1.12.1 
    expo-system-ui: ~3.0.4 => 3.0.4 
    expo-web-browser: ~13.0.3 => 13.0.3 
    jest: ^29.2.1 => 29.7.0 
    jest-expo: ~51.0.1 => 51.0.1 
    react: 18.2.0 => 18.2.0 
    react-dom: 18.2.0 => 18.2.0 
    react-native: 0.74.1 => 0.74.1 
    react-native-gesture-handler: ~2.16.1 => 2.16.2 
    react-native-get-random-values: ^1.11.0 => 1.11.0 
    react-native-reanimated: 3.10.0 => 3.10.0 
    react-native-safe-area-context: ^4.10.1 => 4.10.1 
    react-native-screens: 3.31.1 => 3.31.1 
    react-native-url-polyfill: ^2.0.0 => 2.0.0 
    react-native-web: ~0.19.10 => 0.19.11 
    react-test-renderer: 18.2.0 => 18.2.0 
    tsx: ^4.9.3 => 4.9.3 
    typescript: ^5.4.5 => 5.4.5 (4.4.4, 4.9.5)
  npmGlobalPackages:
    eas-cli: 7.4.0
    npm: 10.7.0
    yarn: 1.22.22

Describe the bug

Follwing the official Gen 2 Guide to configure a Lambda function with an Amazon DynamoDB stream as an event source I get the following error during deployment/sandbox:

The CloudFormation deployment has failed.
Caused By: ❌ Deployment failed: Error [ValidationError]: Circular dependency between resources: [auth179371D7, data7552DF31, function1351588B]


Resolution: Find more information in the CloudFormation AWS Console for this stack.

The error happens when I add the Event Source according to the official Docs:

import { defineBackend } from '@aws-amplify/backend';
import { StartingPosition } from 'aws-cdk-lib/aws-lambda';
import { DynamoEventSource } from 'aws-cdk-lib/aws-lambda-event-sources';
import { auth } from './auth/resource';
import { data } from './data/resource';
import { myDynamoDBFunction } from './functions/dynamoDB-function/resource';

const backend = defineBackend({
  auth,
  data,
  myDynamoDBFunction,
});

const eventSource = new DynamoEventSource(
  backend.data.resources.tables['Todo'],
  {
    startingPosition: StartingPosition.LATEST,
  },
);

backend.myDynamoDBFunction.resources.lambda.addEventSource(eventSource); // <-- this causes the Circular dependency

Edit: I was concerned my Auth preTokenGeneration handler caused this, but it dit not. When I remove preTokenGeneration from triggers, I still get the error:

The CloudFormation deployment has failed.
Caused By: ❌ Deployment failed: Error [ValidationError]: Circular dependency between resources: [data7552DF31, function1351588B]

So the circular dependency definitely results from the fact that the backend resource table is referenced in the eventSource and then being injected back into the backend object. Design flaw?

const eventSource = new DynamoEventSource(
  backend.data.resources.tables['Todo'], // <- reference from backend
  {
    startingPosition: StartingPosition.LATEST,
  },
);

backend.myDynamoDBFunction.resources.lambda.addEventSource(eventSource); // <- inject back into backend

Expected behavior

It should be possible to add the event source for the lambda according to the docs to invoke the lambda by DynamoDB Stream.

Reproduction steps

Just follow your own official guide: https://docs.amplify.aws/react/build-a-backend/functions/examples/dynamo-db-stream/

Code Snippet

// Put your code below this line.

Log output

// Put your logs below this line

aws-exports.js

No response

Manual configuration

No response

Additional configuration

No response

Mobile Device

No response

Mobile Operating System

No response

Mobile Browser

No response

Mobile Browser Version

No response

Additional information and screenshots

No response

[MattWlodarski]

Hey guys, I get the same error:

The CloudFormation deployment has failed. Caused By: ❌ Deployment failed: Error [ValidationError]: Circular dependency between resources: [auth179371D7, data7552DF31, function1351588B]

Here is the code I added to backend.ts:

const SMSResponseEventSource = new DynamoEventSource(backend.data.resources.tables['SMSResponse'], {
  startingPosition: StartingPosition.LATEST,
});
const EmailResponseEventSource = new DynamoEventSource(
  backend.data.resources.tables['EmailResponse'],
  {
    startingPosition: StartingPosition.LATEST,
  },
);
const VoiceCallResponseEventSource = new DynamoEventSource(
  backend.data.resources.tables['VoiceCallResponse'],
  {
    startingPosition: StartingPosition.LATEST,
  },
);

backend.OnAppointmentReminderDeliveryHandler.resources.lambda.addEventSource(
  SMSResponseEventSource,
);
backend.OnAppointmentReminderDeliveryHandler.resources.lambda.addEventSource(
  EmailResponseEventSource,
);
backend.OnAppointmentReminderDeliveryHandler.resources.lambda.addEventSource(
  VoiceCallResponseEventSource,
);

I'm hoping this gets fixed soon so I don't have to set this all up manually.

[MattWlodarski]

I also get the same error when trying to set up environment variables. The weird things with this is that the error won't always happen immediately. Instead, I'll start getting the error after making other unrelated changes to my sandbox. Then I have to comment out the environment variable code in my backend.ts file to get rid of the circular dependency.

The CloudFormation deployment has failed. Caused By: ❌ Deployment failed: Error [ValidationError]: Circular dependency between resources: [auth179371D7, data7552DF31, function1351588B]

Commenting out these lines in backend.ts resolves the circular dependency:

const copyDynamoDBTableLambdaFunction = backend.CopyDynamoDBTableHandler.resources
  .lambda as Function;
copyDynamoDBTableLambdaFunction.addEnvironment('APP_SYNC_API_ID', backend.data.apiId);

[palpatim]

Assigning to @AnilMaktala to try & repro -- I'm not able to get this error to occur.

[LukaASoban]

Just wanted to pop in to say that I also am getting the same error when trying to add in event source for dynamodb streams. Same issue circular dependency between auth, data, and function which doesn't make any sense to me since my lambda needs the table but my table doesn't need my lambda?

I also get the same issue as @MattWlodarski when setting up env variables.

[jgo80]

The issue should be marked as bug rather than as question. Otherwise @AnilMaktala @palpatim proof your guide works correctly.

[LukaASoban]

@jgo80 do you have a workaround by any chance for now?

[jgo80]

@jgo80 do you have a workaround by any chance for now?

I don't 😞 Just waiting for an answer as a desperate customer moving from Gen1 to Gen2...

[MattWlodarski]

I found a similar issue with a resolution: https://github.com/aws-amplify/amplify-backend/issues/1552

I'm not sure if it exactly applies to our situation because I didn't quite follow the recommended solution. Let me know if it helps you guys.

[LukaASoban]

Thanks @MattWlodarski I will take a look at this today and give it a try and then update this thread

[ggj0418]

Yeah the code provided in the document operates normally when only event mapping is used, but a circular dependency error occurs as soon as data custom query is used. So if you are using a custom query (both query and mutation), you have to manually make a separate cdk stack for the streaming target table and set the event mapping and policy on that stack.

As @MattWlodarski mentioned, i wrote down that issue and found the solution. the well-worked code is on my repository (you can find it on that mentioned issue), so you might be able to refer it.

[LukaASoban]

I had some triggers in the auth resource. If I remove them, it removes the circular dependency.

triggers: {
    customMessage,
    postConfirmation,
    preSignUp,
    defineAuthChallenge,
    createAuthChallenge,
    verifyAuthChallengeResponse: verifyAuthChallenge,
  },

[LukaASoban]

@ggj0418 @MattWlodarski do you either of you have any auth triggers?

[LukaASoban]

import { defineAuth, secret } from "@aws-amplify/backend";
import { customMessage } from "./custom-message/resource";
import { postConfirmation } from "./post-confirmation/resource";
import { preSignUp } from "./pre-signup/resource";
import { defineAuthChallenge } from "./define-auth-challenge/resource";
import { createAuthChallenge } from "./create-auth-challenge/resource";
import { verifyAuthChallenge } from "./verify-auth-challenge/resource";

/**
 * Define and configure your auth resource
 * @see https://docs.amplify.aws/gen2/build-a-backend/auth
 */
export const auth = defineAuth({
  loginWith: {
    email: true,
    phone: true,
  },
  triggers: {
    customMessage,
    postConfirmation,
    preSignUp,
    defineAuthChallenge,
    createAuthChallenge,
    verifyAuthChallengeResponse: verifyAuthChallenge,
  },
});

[ggj0418]

@LukaASoban hm... no i do not have any auth trigger. did you add that trigger with custom query or something else component of amplify gen2 backend?

[ggj0418]

@LukaASoban on another project, im using that auth trigger with data custom queries, but it does not have a circular error

[LukaASoban]

@AnilMaktala here is the min code required for me to reproduce

/**
 * @see https://docs.amplify.aws/react/build-a-backend/ to add storage, functions, and more
 */
const backend = defineBackend({
  auth,
  data,
  dynamodbStreamPlaces,
});

const eventSource = new DynamoEventSource(
  backend.data.resources.tables["Place"],
  {
    startingPosition: StartingPosition.LATEST,
  }
);

backend.dynamodbStreamPlaces.resources.lambda.addEventSource(eventSource);

and then my auth resource.ts

/**
 * Define and configure your auth resource
 * @see https://docs.amplify.aws/gen2/build-a-backend/auth
 */
export const auth = defineAuth({
  loginWith: {
    email: true,
    phone: true,
  },
  triggers: {
    customMessage,
  },
});

and my data resource.ts

import { type ClientSchema, a, defineData } from "@aws-amplify/backend";

const schema = a.schema({
  Place: a
    .model({
      id: a.id(),
      globalId: a.string().required(),
      name: a.string().required(),
      description: a.string(),
      coordinates: a.customType({
        latitude: a.float().required(),
        longitude: a.float().required(),
      }),
      address: a.string().required(),
      createdAt: a.datetime(),
      updatedAt: a.datetime(),
      defaultPhotoId: a.id(),
    })
    .authorization((allow) => [
      allow.guest().to(["read"]),
      allow.authenticated().to(["read", "create"]),
    ]),
});

export type Schema = ClientSchema<typeof schema>;

export const data = defineData({
  schema,
  authorizationModes: {
    defaultAuthorizationMode: "userPool",
  },
});

and my customMessage function

import { defineFunction } from "@aws-amplify/backend";

export const customMessage = defineFunction({
  name: "custom-message",
});

import type { CustomMessageTriggerHandler } from "aws-lambda";

export const handler: CustomMessageTriggerHandler = async (event) => {
  return event;
};

[ggj0418]

@LukaASoban I think that dynamoDB event mapping is the point. here is my suggestion.

const backend = defineBackend({
  auth,
  //preSignUp,
  //storage,
  data,
  //getUsersByEmailOrPhoneFunction,
  //createAuthChallenge,
  //checkIfCognitoUserExistsFunction,
  //onUpload,
  dynamodbStreamPlaces,
});

const placeTable = backend.data.resources.tables["Place"]
backend.dynamodbStreamPlaces.resources.lambda.role?.attachInlinePolicy(
    new Policy(
        Stack.of(placeTable),
        "DynamoDBPolicy",
        {
            statements: [
                new PolicyStatement({
                    effect: Effect.ALLOW,
                    actions: [
                        "dynamodb:DescribeStream",
                        "dynamodb:GetRecords",
                        "dynamodb:GetShardIterator",
                        "dynamodb:ListStreams",
                    ],
                    resources: ["*"],
                }),
            ],
        }
    )
);

new EventSourceMapping(
    Stack.of(placeTable),
    "test",
    {
        target: backend.dynamodbStreamPlaces.resources.lambda,
        eventSourceArn: placeTable.tableStreamArn,
        startingPosition: StartingPosition.LATEST,
    }
);

[ggj0418]

Oh i forgot to tell the resolved code's branch. here is the branch. it is not a main. i'm sorry for the missing description.

[LukaASoban]

@ggj0418 Thanks for your input here. I seem to be getting this error when I try that:

CREATE_FAILED | AWS::Lambda::EventSourceMapping | data/amplifyData/Place/DynamodbStreamPlacesEventSourceMapping (DynamodbStreamPlacesEventSourceMappingE2426243) Resource handler returned message: "Invalid request provided: Stream arn:aws:dynamodb:us-east-2:*********:table/Place-2xgy7oir4fa5tndcygcxeusazu-NONE/stream/2024-05-30T20:36:57.557 is Disabled. You cannot create a lambda mapping on a stream that is Disabled. (Service: Lambda, Status Code: 400, Request ID: 1578ca62-c2ba-4039-ab59-15f48b13ba0e)" (RequestToken: ea51563f-a008-1cb5-0938-f9869001ab6f, HandlerErrorCode: InvalidRequest)

Not sure why my lambda stream is disabled

[ggj0418]

@LukaASoban hm... could i see your dynamdbStreamPlaces' handler?

[LukaASoban]

Looks exactly like in the docs:

import type { DynamoDBStreamHandler } from "aws-lambda";
import { Logger } from "@aws-lambda-powertools/logger";

const logger = new Logger({
  logLevel: "INFO",
  serviceName: "dynamodb-stream-handler",
});

export const handler: DynamoDBStreamHandler = async (event) => {
  for (const record of event.Records) {
    logger.info(`Processing record: ${record.eventID}`);
    logger.info(`Event Type: ${record.eventName}`);

    if (record.eventName === "INSERT") {
      // business logic to process new records
      logger.info(`New Image: ${JSON.stringify(record.dynamodb?.NewImage)}`);
    }
  }
  logger.info(`Successfully processed ${event.Records.length} records.`);
  
  return {
    batchItemFailures: [],
  };
};

[LukaASoban]

@ggj0418 So I deleted all my resources and re-created them in my sandbox and it worked! Thanks for the work-around.

I still get a circ dep when I add in env vars though

[ggj0418]

@LukaASoban oh yeah sandbox's hotswap is not perfectly worked... do you talk about the amplify env vars?

[heclon]

How come the official docs publishes a procedure that clearly causes this issue? Does no one checks what is being published?

This should be marked as an issue and fixed.

[LukaASoban]

@heclon are you also getting this same error?

[MattWlodarski]

Hey @LukaASoban I also had an auth trigger as well. I appreciate you guys explaining the workaround but I agree with @heclon. This should be marked as an issue and fixed asap. I can't believe they haven't caught this already. They must just test their code in the simplest possible ways. It doesn't take much to trigger either of these circular dependencies...

[AnilMaktala]

HI @jgo80, We are able to replicate this issue and marking this as bug for the team to evaluate further.

[leonanderson88]

I'm facing the same issue.

• Data
• Lambda function as a custom mutation
• Lambda function attempting to set up DynamoDBStream as event source and send ses notifications.

I have tried ggj0418 solution but getting errors regarding permissions. Still haven't got the work around to work. Is there any other way to get a DynamoDBStream as an event source.

This is delaying deployment of a production site.

Is there any update on the time frame for this bug to be fixed?

[ggj0418]

@leonanderson88 Does the same error occur even if you comment that ses part?

[LukaASoban]

EDIT The below code is probably not necessary for most. For some reason, the above work around was complaining about the streamARN being disabled for me. If you are having that issue then you can use the below, otherwise it's not needed

@leonanderson88 The below works - here is my code to get this to work to break the circular dependency (you can reduce the IAM role scopes as needed I just gave it all permissions to see if it works)

You basically need to create a custom resource that is a lambda that runs, uses the AWS SDK to get the streamARN for the table and then returns it. You can modify it to include as many tables as you want.

One thing to note is that I am not sure how it works when deleting resources and then recreating them. Upon that first pass it might not find any stream arns for the table since it's possible it wasn't created yet. In that case you would need to do another update to your CFN stack.

IMO this is one of the biggest bugs / issue with Gen 2, the circular dependencies are hard to break, passing table names to functions should not cause circ deps

/* backend.ts */
const userProfileTable = backend.data.resources.tables["UserProfile"];

const dynamodbStreamingLambda =
  backend.dbToOpensearchStreamingFunction.resources.lambda;
dynamodbStreamingLambda.addToRolePolicy(
  new iam.PolicyStatement({
    sid: "AllowStreaming",
    actions: ["dynamodb:*"],
    resources: ["*"],
  }),
);

// getStreamARN Lambda IAM Policy
const getStreamArnsLambdaPolicy = new iam.PolicyStatement({
    actions: ["dynamodb:*"],
    resources: ["*"],
  });


// Create a stack for the custom resources
const customResourceStack = backend.createStack("CustomResourceStack");

// Create a custom resource to init the event mappings (we need to get the stream ARNs)
const getStreamArnsLambda = new lambda.Function(
  customResourceStack,
  "GetStreamArnsLambda",
  {
    runtime: lambda.Runtime.NODEJS_LATEST,
    code: lambda.Code.fromAsset(
      "./amplify/custom/trigger-functions/get-dynamodb-stream-arns",
    ),
    handler: "getStreamArns.handler",
    timeout: Duration.seconds(30),
    initialPolicy: [getStreamArnsLambdaPolicy],
  },
);

// Create a custom resource to init the event mappings (we need to get the stream ARNs for each table)
const streamArnsProvider = new customResources.Provider(
  customResourceStack,
  "StreamArnsProvider",
  {
    onEventHandler: getStreamArnsLambda,
  },
);
const streamArns = new CustomResource(customResourceStack, "StreamArns", {
  serviceToken: streamArnsProvider.serviceToken,
  properties: {
    TableNames: [
      userProfileTable.tableName,
    ], // Add more tables here as we need
  },
});

// Ensure the custom resource depends on the DynamoDB tables creation
streamArns.node.addDependency(userProfileTable);


// Event source mapping for the User Profile table
const userProfileStreamingEventSourceMapping = new lambda.EventSourceMapping(
  customResourceStack,
  "UserProfileStreamingEventSourceMapping",
  {
    target: backend.dbToOpensearchStreamingFunction.resources.lambda,
    eventSourceArn: streamArns.getAttString("UserProfile"),
    startingPosition: StartingPosition.LATEST,
  },
);
userProfileStreamingEventSourceMapping.node.addDependency(streamArns);

and then in my getStreamArns lambda

/* GetStreamArns lambda function */

const {
  DynamoDBClient,
  DescribeTableCommand,
} = require("@aws-sdk/client-dynamodb");

const client = new DynamoDBClient({ region: process.env.AWS_REGION });

exports.handler = async (event) => {
  const tableNames = event.ResourceProperties.TableNames;

  try {
    const streamArns = {};
    for (const tableName of tableNames) {
      const command = new DescribeTableCommand({ TableName: tableName });
      const data = await client.send(command);
      const baseTableName = tableName.split("-")[0]; // Extract the base name
      streamArns[baseTableName] = data.Table.LatestStreamArn || "";
    }

    console.log("Stream ARNs:", JSON.stringify(streamArns)); // Log the returned ARNs

    return {
      PhysicalResourceId: tableNames.join(","),
      Data: streamArns,
    };
  } catch (error) {
    console.error("Error:", error);

    return {
      PhysicalResourceId: tableNames.join(","),
      Data: {
        Error: error.message,
      },
    };
  }
};