String values in ExpressionAttributeValue do not support emoji

[a90120411]

Describe the bug

When the string value in ExpressionAttributeValues contains emoji, UpdateCommand and PutCommand can't save data normally.

Your environment

SDK version number

@aws-sdk/[email protected]

Is the issue in the browser/Node.js/ReactNative?

Browser

Details of the browser/Node.js/ReactNative version

Chrome: 100.0.4896.127

Steps to reproduce

function initDynamoDBClient() {
  dynamoDBClient = new DynamoDBClient({
    region: "ap-east-1",
    credentials: {
      accessKeyId: AWSAccessKeyId,
      secretAccessKey: AWSSecretAccessKey,
    },
    endpoint: "https://dynamodb.ap-east-1.amazonaws.com",
  });
  const marshallOptions = {
    convertEmptyValues: false,
    removeUndefinedValues: false,
    convertClassInstanceToMap: false
  };
  const unmarshallOptions = {
    wrapNumbers: false
  };
  const translateConfig = { marshallOptions, unmarshallOptions };
  ddbDocClient = DynamoDBDocumentClient.from(dynamoDBClient, translateConfig);
  console.log("initDynamoDBClient finished");
}

async function updateHistoryItem(item) {
  const params = {
    TableName: tbName,
    Key: {
      url: item.url,
    },
    ExpressionAttributeValues: {
      ":read_state": item.read_state || 0,
      ":title": item.title,   //title is a string , include emoji.
    },
    UpdateExpression: `SET 
    title = :title,
    read_state = :read_state`,
  };
  return updateItemFromDynamoDB(params);
}

async function updateItemFromDynamoDB(params) {
  return sendCommandToDynamoDB(new UpdateCommand(params));
}

async function putItemFromDynamoDB(params) {
  return sendCommandToDynamoDB(new PutCommand(params));
}

async function sendCommandToDynamoDB(command) {
  try {
    const data = await ddbDocClient.send(command);
    console.log("AWS DynamoDB send command -- succeeded");
    return data;
  } catch (err) {    
    console.error(
      "AWS DynamoDB send command -- failed:" + "\n" + JSON.stringify(err)
    );
    return null;
  }
}

Observed behavior

Exception was thrown

failed: {"__type":"com.amazon.coral.service#InvalidSignatureException","name":"InvalidSignatureException","$fault":"client","$metadata":{"httpStatusCode":400,"requestId":"ML8R85HMGCI4FL4KQA4NKHPG5VVV4KQNSO5AEMVJF66Q9ASUAAJG","attempts":3,"totalRetryDelay":477}}

InvalidSignatureException: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

Expected behavior

The string value containing the emoji was successfully saved or provide a function for encoding.

[RanVaknin]

@a90120411 Hi,

Im trying to replicate your code but it's missing a lot of things like the correct import statements, variable declarations. and also it would be very helpful if you have the schema code you used to set up the Ddb table with.

Thanks!

[a90120411]

@a90120411 Hi,

Im trying to replicate your code but it's missing a lot of things like the correct import statements, variable declarations. and also it would be very helpful if you have the schema code you used to set up the Ddb table with.

Thanks!

Sorry, I added these information.

const { DynamoDBClient } = require("@aws-sdk/client-dynamodb");
import {
  DynamoDBDocumentClient,
  PutCommand,
  GetCommand,
  UpdateCommand,
} from "@aws-sdk/lib-dynamodb";

let dynamoDBClient = null;
let ddbDocClient = null;

[RanVaknin]

Hi @a90120411,

I am not able to replicate your error because I'm missing two things:

1. your schema and table creation code.
2. the item object you're feeding your function

I suspect this error is coming from the way you are referring to the values in your item object, but again this is only an assumption since I don't have your full code. Please take a look at this Stack Overflow issue. I know that the original poster did not experience the same issue as you but you might be able to draw some useful information from the solution.

Despite the missing information, I was able to :

• Create a ddb Table
• Putting an item in the table with the following fields:

Partition key: Season (Number): 3
Sort key: Episode (Number): 10
Title (String): "Benihanna Christmas"
Rank (String): "⭐⭐⭐⭐"

• Updating the same item by identifying it using partition and sort key, and passing the new updated value (five stars instead of 4):

Rank (String): "⭐⭐⭐⭐⭐"

Conclusion: The SDK's UpdateCommand and PutCommand do support emojis in ExpressionAttributeValues. Please follow the tutorial I linked in each of my steps.

Let me know if you have any other issues.

[github-actions[bot]]

This issue has not received a response in 1 week. If you still think there is a problem, please leave a comment to avoid the issue from automatically closing.

[a90120411]

I will add some info.

[RanVaknin]

@a90120411 Have you tried following the steps I've mentioned in my walkthrough?

[a90120411]

@RanVaknin
Thanks for the information, unfortunately it didn't solve the problem.

I created a test table: tb_demo, It contains two fields: id:String(key) and title:String. Execute the following code( I use webpack to package and run in the browser): If I remove the emoji symbol in the title value, everything works fine.

This problem only happens in browser environment. It works fine in Node.js environment and PartiQL editor.

const { DynamoDBClient } = require("@aws-sdk/client-dynamodb");
const {
  DynamoDBDocumentClient,
  PutCommand,
  GetCommand,
} = require("@aws-sdk/lib-dynamodb");

const tbName = "tb_demo";
let dynamoDBClient = null;
let ddbDocClient = null;

function initDynamoDBClient() {
  dynamoDBClient = new DynamoDBClient({
    region: "ap-east-1",
    credentials: {
      accessKeyId: "XXXXXXX",
      secretAccessKey: "XXXXXX",
    },
    endpoint: "https://dynamodb.ap-east-1.amazonaws.com",
  });
  const marshallOptions = {    
    convertEmptyValues: false,    
    removeUndefinedValues: false,
    convertClassInstanceToMap: false,
  };
  const unmarshallOptions = {    
    wrapNumbers: false,
  };
  const translateConfig = { marshallOptions, unmarshallOptions };
  ddbDocClient = DynamoDBDocumentClient.from(dynamoDBClient, translateConfig);
  console.log("initDynamoDBClient finished");
}

async function putHistoryItem(item) {
  const params = {
    TableName: tbName,
    Item: item,
  };
  return putItemFromDynamoDB(params);
}

async function getDetailHistoryItem(id) {
  const params = {
    TableName: tbName,
    Key: {
      id: id,
    },
  };
  return getItemFromDynamoDB(params);
}

async function getItemFromDynamoDB(params) {
  return sendCommandToDynamoDB(new GetCommand(params));
}

async function putItemFromDynamoDB(params) {
  return sendCommandToDynamoDB(new PutCommand(params));
}

async function sendCommandToDynamoDB(command) {
  try {
    const data = await ddbDocClient.send(command);
    console.log("AWS DynamoDB send command -- succeeded");
    return data;
  } catch (err) {
    console.error(err);
    console.error(
      "AWS DynamoDB send command -- failed:" + "\n" + JSON.stringify(err)
    );
    return null;
  }
}
//run
initDynamoDBClient();
putHistoryItem( {id: "1", Title: "😄Emoji"});
getDetailHistoryItem("1").then((data) => {
  console.log(data);
});

Exception information:

serviceWorker.js:27559 InvalidSignatureException: The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.
    at file:///Users/source/frontend/WebHistoryTool/aws-dynamodb/dist/serviceWorker.js:14125:68
    at step (file:///Users/source/frontend/aws-dynamodb/dist/serviceWorker.js:694:23)
    at Object.next (file:///Users/source/frontend/aws-dynamodb/dist/serviceWorker.js:675:53)
    at fulfilled (file:///Users/source/frontend/aws-dynamodb/dist/serviceWorker.js:665:58)
sendCommandToDynamoDB @ serviceWorker.js:27559
await in sendCommandToDynamoDB (async)
putItemFromDynamoDB @ serviceWorker.js:27545
putHistoryItem @ serviceWorker.js:27510
(anonymous) @ serviceWorker.js:27493
(anonymous) @ serviceWorker.js:27576
(anonymous) @ serviceWorker.js:27578

serviceWorker.js:27560 AWS DynamoDB send command -- failed:
{"__type":"com.amazon.coral.service#InvalidSignatureException","name":"InvalidSignatureException","$fault":"client","$metadata":{"httpStatusCode":400,"requestId":"GJOQ4811OATTBTAHBM75R0F9GRVV4KQNSO5AEMVJF66Q9ASUAAJG","attempts":3,"totalRetryDelay":326}}

[a90120411]

"The request signature we calculated does not match the signature you provided." Why is the signature error displayed? Maybe this is a bug due to encoding?

[RanVaknin]

@a90120411 ,

Thanks for helping me reproduce this issue. This is incredibly odd! It only breaks when bundled with Webpack but only for certain emojis. Requests with ⭐emojis are sent fine, but the 😄 emoji that you used was not supported. It might be an encoding issue. I assigned the issue to one of the developers on the team but since this is an edge case this would probably take low priority.

Please consider this hacky solution to unblock you while we work on a fix. One of the things you could do is encode the emoji, and when pulling data from AWS decode the emoji to display correctly in your application. Something like:

putItem.js

//initialize client, client.send(putItem....)

ExpressionAttributeValues: {
        ":rank": { S: "utf8.encode("😄") } // would save it as "8J+YhA=="
    }

getItem.js

// const item = await getItem(key)

console.log(utf8.decode(item.rank) // would turn into "😄"

[RanVaknin]

@trivikr , I have confirmed this is not a service related issue by doing the same action from the AWS CLI:

aws dynamodb update-item \
    --table-name "TEST_TABLE" \
    --key file://test_folder/Key.json \
    --update-expression "SET #rank = :rank" \
    --expression-attribute-names file://test_folder/ExpressionAttributeNames.json \
    --expression-attribute-values file://test_folder/ExpressionAttributeValues.json \
    --region us-east-1 \
    

This is a concise version needed to repro the issue.

// ====> needs to be bundled with Webpack <====

import { UpdateItemCommand } from "@aws-sdk/client-dynamodb";
import { ddbClient } from "../libs/ddbClient.js";

export const params = {
    TableName: "TEST_TABLE",
    Key: {
        Season: { N: "3" } ,
        Episode: { N: "10"} ,
    },
    ExpressionAttributeNames:{
        "#rank": "Rank"
    },
    UpdateExpression: "set #rank = :rank",
    ExpressionAttributeValues: {
        ":rank": { S: "😄" }
    },
    ReturnValues: "ALL_NEW"
};
export const run = async () => {
    try {
        const data = await ddbClient.send(new UpdateItemCommand(params));
        console.log(data);
        return data;
    } catch (err) {
        console.error(err);
    }
};
run();

// ====> needs to be bundled with Webpack <====

Logging middleware (build stage and finalizeRequest stage)

additionally, please take a look at this issue. Perhaps its related.

[trivikr]

I'm able to reproduce the issue using 😄 emoji only in AWS SDK for JavaScript (v3)

To reproduce create an example DynamoDB table, or using the code below:

createTable

import { DynamoDB } from "@aws-sdk/client-dynamodb";

const TableName = "aws-sdk-js-v3-3559";

const params = {
  TableName,
  AttributeDefinitions: [{ AttributeName: "id", AttributeType: "S" }],
  KeySchema: [{ AttributeName: "id", KeyType: "HASH" }],
  BillingMode: "PAY_PER_REQUEST",
};

const region = "us-west-2";
const client = new DynamoDB({ region });
client.createTable(params);

Use the following patch file in https://github.com/aws-samples/aws-sdk-js-tests

put-item-with-emoji.txt

Code runs successfully in Node.js as follows:

$ yarn start:node
...
Data returned by v2:
{
  "Item": {
    "id": {
      "S": "Emoji from v2 😄"
    }
  }
}

Data returned by v3:
{
  "$metadata": {
    "httpStatusCode": 200,
    "requestId": "B10RV813CNJFVV0EMVC8P987JRVV4KQNSO5AEMVJF66Q9ASUAAJG",
    "attempts": 1,
    "totalRetryDelay": 0
  },
  "Item": {
    "id": {
      "S": "Emoji from v3 😄"
    }
  }
}

Code fails in v3 on browser with the following error in browser console:

The request signature we calculated does not match the signature you provided. Check your AWS Secret Access Key and signing method. Consult the service documentation for details.

Screenshot

[trivikr]

It appears to be an issue with Content-Length in the browser request.

Here is a patch file with a middleware which prints headers put-item-with-emoji-and-middleware.txt

With the text Hello World!, here is the diff between Node.js and browser

string no emoji diff

Rename .txt to .html to view after download: string-no-emoji-diff.txt

With the text Hello World!😄 the content-length in Node.js is 73, while that in browser is 75.

string with emoji diff

Rename .txt to .html to view after download: string-emoji-diff.txt

[trivikr]

This bug was introduced when usage of Blob was replaced with manual computation of length in https://github.com/aws/aws-sdk-js-v3/pull/1384

[trivikr]

Fix posted in https://github.com/aws/aws-sdk-js-v3/pull/3866

[github-actions[bot]]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs and link to relevant comments in this thread.