Unmarshing DynamoStream events

[maxm450]

Describe the bug

TS types seems to cause an issue when unmarshing a dynamoStream

Argument of type '{ [key: string]: AWSLambda.AttributeValue; }' is not assignable to parameter of type '{ [key: string]: import("/node_modules/@aws-sdk/client-dynamodb/dist/types/models/models_0").AttributeValue; }'.

Your environment

SDK version number

@aws-sdk/[email protected]

Steps to reproduce

import { DynamoDBStreamEvent } from 'aws-lambda'
import { unmarshall } from "@aws-sdk/util-dynamodb"

export const handler =  async (event) => {
  for (const record of event.Records) {
    if (record.dynamodb?.NewImage) {
      const streamItem = record.dynamodb?.NewImage
      const item = unmarshall(streamItem)
      console.log(item)
    }
  }
}

Observed behavior

receiving typing issue for AttributeValue type.

Expected behavior

should be able to unmarshall a Dynamo object, wherever the object comes from a stream event or a getItem action from the DynamoClient

[monken]

unmarshall will also not convert binary attributes properly. The DynamoDB stream will provide a base64 encoded presentation of the binary attribute and unmarshall does not base64-decode the value.

[sekhavati]

I'm still seeing this issue in 3.34.0.

Interestingly there are no type errors if my package only contains @aws-sdk/util-dynamodb. But as soon as I install @aws-sdk/client-dynamodb the aforementioned TypeScript errors pop up.

From my brief testing with unmarshall it behaved as expected prior to installing @aws-sdk/client-dynamodb , so as a workaround I've set a @ts-ignore on the problematic line of code for now.

[ajredniwja]

Hey everyone, apologies the issue fell out of queue. A lot of changes and features were added to the latest package. Is it still an persisting issue with the latest version. I was able to use unmarshall correctly used: https://www.npmjs.com/package/@aws-sdk/lib-dynamodb.

[faridnsh]

Yes, this is still an issue as the types are not compatible. More info about it in DefinitelyTyped/DefinitelyTyped#51331

[nikiitat]

Any updates on this one? I think this is also related to: https://github.com/DefinitelyTyped/DefinitelyTyped/issues/51331

[obones]

unmarshall will also not convert binary attributes properly. The DynamoDB stream will provide a base64 encoded presentation of the binary attribute and unmarshall does not base64-decode the value.

I'm facing the same behavior here and I'm not sure what to do here. Should this be reported in a separate issue?

[jimmyn]

Any update on this? I'm facing the same issue with the latest version

[zshzbh]

Hey everyone,

I want to provide this workaround so you can decode the binary attributes.

import { DynamoDBClient } from "@aws-sdk/client-dynamodb";
import { unmarshall } from "@aws-sdk/util-dynamodb";

const dynamoClient = new DynamoDBClient({ region: "us-west-2" });

function customUnmarshall(item) {
  const unmarshalled = unmarshall(item);
  
  for (const [key, value] of Object.entries(unmarshalled)) {
    if (item[key] && item[key].B) {
      // Decode the base64 string to a UTF-8 string
      unmarshalled[key] = Buffer.from(item[key].B, 'base64').toString('utf-8');
    }
  }
  
  return unmarshalled;
}

export const handler = async (event) => {
  for (const record of event.Records) {
    if (record.dynamodb?.NewImage) {
      const streamItem = record.dynamodb.NewImage;
      const item = customUnmarshall(streamItem);
      console.log("Unmarshalled item (including decoded binary attributes):", item);
      
      for (const [key, value] of Object.entries(item)) {
        if (typeof value === 'string' && item[key] && item[key].B) {
          console.log(`Decoded binary attribute ${key}:`, value);
        }
      }
    }
  }
};

[zshzbh]

Using the workaround above, I used the following test case of DynamoDB object and it's working :

{
  "Records": [
    {
      "eventID": "1",
      "eventName": "INSERT",
      "eventVersion": "1.0",
      "eventSource": "aws:dynamodb",
      "awsRegion": "us-west-2",
      "dynamodb": {
        "Keys": {
          "Id": {
            "N": "101"
          }
        },
        "NewImage": {
          "Id": {
            "N": "101"
          },
          "Name": {
            "S": "John Doe"
          },
          "BinaryData": {
            "B": "SGVsbG8gV29ybGQ="
          }
        },
        "SequenceNumber": "111",
        "SizeBytes": 26,
        "StreamViewType": "NEW_AND_OLD_IMAGES"
      },
      "eventSourceARN": "arn:aws:dynamodb:us-west-2:123456789012:table/YourTableName/stream/2023-05-11T00:00:00.000"
    },
    {
      "eventID": "2",
      "eventName": "MODIFY",
      "eventVersion": "1.0",
      "eventSource": "aws:dynamodb",
      "awsRegion": "us-west-2",
      "dynamodb": {
        "Keys": {
          "Id": {
            "N": "102"
          }
        },
        "NewImage": {
          "Id": {
            "N": "102"
          },
          "Name": {
            "S": "Jane Smith"
          },
          "BinaryData": {
            "B": "SGVsbG8gQWdhaW4="
          }
        },
        "SequenceNumber": "222",
        "SizeBytes": 28,
        "StreamViewType": "NEW_AND_OLD_IMAGES"
      },
      "eventSourceARN": "arn:aws:dynamodb:us-west-2:123456789012:table/YourTableName/stream/2023-05-11T00:00:00.000"
    }
  ]
}

This is the log I got "

START RequestId: 1c248220-6a2f-49e0-a15b-0d32ac45d714 Version: $LATEST
2024-10-29T16:03:16.831Z	1c248220-6a2f-49e0-a15b-0d32ac45d714	INFO	Unmarshalled item (including decoded binary attributes): { Id: 101, Name: 'John Doe', BinaryData: 'Hello World' }
2024-10-29T16:03:16.833Z	1c248220-6a2f-49e0-a15b-0d32ac45d714	INFO	Unmarshalled item (including decoded binary attributes): { Id: 102, Name: 'Jane Smith', BinaryData: 'Hello Again' }
END RequestId: 1c248220-6a2f-49e0-a15b-0d32ac45d714
REPORT RequestId: 1c248220-6a2f-49e0-a15b-0d32ac45d714	Duration: 303.34 ms	Billed Duration: 304 ms	Memory Size: 128 MB	Max Memory Used: 87 MB	

@jimmyn, Could you please help me understand what issue you are facing now? Do you have any minimal code reproduction and error msg?

Thanks!

[zshzbh]

Just talked with the team, it seems the issue is caused by the type mismatch.

Please vote or reply if you are encountering this issue with aws-lambda pkg installed

[jimmyn]

Just talked with the team, it seems the issue is caused by the type mismatch.

Please vote or reply if you are encountering this issue with aws-lambda pkg installed

Yes, this is exactly what happens.

[cbudau]

I investigated this issue and found that the root cause lies in the aws-lambda package, which defines its own AttributeValue type instead of using the one provided by @aws-sdk/client-dynamodb. This results in an unnecessary type conflict when working with DynamoDB Streams in AWS Lambda, causing TypeScript errors when attempting to unmarshall stream events.

Key Findings:

• The issue is not caused by @aws-sdk/client-dynamodb, but rather by aws-lambda maintaining a separate AttributeValue definition.
• Fixing this in the SDK is not the right approach, as it would require importing aws-lambda into @aws-sdk/client-dynamodb, which is not recommended. This would introduce unnecessary dependencies and reduce the SDK’s compatibility across different environments.
• The correct solution is for aws-lambda to update its type definitions to use @aws-sdk/client-dynamodb.AttributeValue instead of defining its own.

Temporary Workaround for Users

Until this is fixed in aws-lambda, affected users can resolve the issue in their Lambda functions by explicitly casting AWSLambda.AttributeValue to @aws-sdk/client-dynamodb.AttributeValue, as shown below:

import { DynamoDBStreamEvent } from 'aws-lambda';
import { unmarshall } from '@aws-sdk/util-dynamodb';
import { AttributeValue } from '@aws-sdk/client-dynamodb';

export const handler = async (event: DynamoDBStreamEvent) => {
  for (const record of event.Records) {
    if (record.dynamodb?.NewImage) {
      // Type casting avoids TypeScript error
      const streamItem = record.dynamodb.NewImage as { [key: string]: AttributeValue };
      const item = unmarshall(streamItem);
      console.log(item);
    }
  }
};

[zshzbh]

Hey @cbudau , you are right.

Thanks for the workaround. It seems that this mismatch is introduced from aws-lambda pkg. I think right now there's 2 workarounds available. The first one is to avoid using aws-lambda pkg, and use @aws-sdk/client-dynamodb pkg instead (as I posted here - https://github.com/aws/aws-sdk-js-v3/issues/2634#issuecomment-2444689575). The second one is what you posted, which is casting AWSLambda.AttributeValue to @aws-sdk/client-dynamodb.AttributeValue.

[zshzbh]

Hey fellows,

aws-lambda is not owned by Amazon or AWS. We are not responsible to update aws-lambda package. You can submit a issue in its own repo https://github.com/awspilot/cli-lambda-deploy.

@aws-sdk/client-dynamodb and @aws-sdk/util-dynamodb are packaged that are owned by AWS. Using these two packages will not cause this issue. Please refer to my comment for sample usage - https://github.com/aws/aws-sdk-js-v3/issues/2634#issuecomment-2444689575

[zshzbh]

I'm closing this issue as there's no issue in AWS owned packages

[github-actions[bot]]

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.