Lambda Otel Layer Adding Too Many SQS MessageAttributes?

[Peter-J-Brown]

I've got a lambda which places messages on an SQS queue, and which is instrumented with open-telemetry. When the lambda places a message on the queue, it attaches two MessageAttributes to the message, but whenever the lambda calls sendMessage(), the lambda otel layer logs the following: aws-sdk instrumentation: cannot set context propagation on SQS/SNS message due to maximum amount of MessageAttributes Note: the traces are not being dropped and the messages are successfully placed on the SQS queue.

What's confusing me here is that I know SQS has a limit of 10 MessageAttributes on a message when raw message delivery is enabled (which it is in this case), but I don't understand how I'm getting anywhere near that limit considering that at the point the lambda calls sendMessage(), the message it passes in only has 2 MessageAttributes. I've also checked the message again once it's been put on the queue, and I can't see any new attributes that have been added. Is the otel lambda layer trying to add 8+ MessageAttributes to the message before it's placed on the SQS queue?

Context: I'm using lambda otel layer arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-nodejs-amd64-ver-1-5-0:2, but I have also tested downgrading to arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-nodejs-amd64-ver-1-5-0:2 and experienced the same issue.

[tsloughter]

Do you have a configured propagator or using the defaults?

I'm not very familiar with JS or TS but maybe something is off with this check it is doing before adding the attributes, https://github.com/open-telemetry/opentelemetry-js-contrib/blob/main/plugins/node/opentelemetry-instrumentation-aws-sdk/src/services/MessageAttributes.ts#L75

[tsloughter]

Also note this appears to be an issue with the SQS instrumentation and unrelated to Lambda. You should open an issue with https://github.com/open-telemetry/opentelemetry-js-contrib.

We'll need to update the dependency in the layer if this is a bug in the AWS SDK instrumentation there, so good to still have this open.

[github-actions[bot]]

This issue was marked stale. It will be closed in 30 days without additional activity.