botocore SQS custom `MessageAttributes` are not being attached to spans

[miketheman]

Summary of problem

Using ddtrace and botocore, we do not see custom MessageAttributes being added to the ddtrace spans.

We added setting custom message attributes via celery kwargs, and can confirm the custom attributed are being emitted to botocore, but are not being expanded or attached to the spans.

Which version of dd-trace-py are you using?

ddtrace==2.10.3

https://github.com/pypi/warehouse/blob/754886cb4e49226dfe75171d2cfec8aff0052ff4/requirements/deploy.txt#L25

Which version of pip are you using?

24.2

Which libraries and their versions are you using?

See https://github.com/pypi/warehouse/blob/754886cb4e49226dfe75171d2cfec8aff0052ff4/requirements/main.txt for full list

How can we reproduce your problem?

Set custom MessageAttributes on a call while being traced https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/sqs/client/send_message.html

For example, "MessageAttributes": {"task_name": {"StringValue": "example_task_name", "DataType": "String"}

What is the result that you get?

Custom MessageAttributes are emitted - but only the _datadog ones get handled.

Here's an example of a logged debug call from botocore in a local development environment:

{"logger": "botocore.endpoint", "level": "DEBUG", "event": "Making request for OperationModel(name=SendMessage) with params: {'url_path': '/', 'query_string': '', 'method': 'POST', 'headers': {'X-Amz-Target': 'AmazonSQS.SendMessage', 'Content-Type': 'application/x-amz-json-1.0', 'User-Agent': 'Boto3/1.34.134 md/Botocore#1.34.134 ua/2.0 os/linux#6.9.8-orbstack-00170-g7b4100b7ced4 md/arch#aarch64 lang/python#3.12.4 md/pyimpl#CPython cfg/retry-mode#legacy Botocore/1.34.134'}, 'body': b'{"QueueUrl": "http://sqs.us-east-1.localstack:4566/000000000000/warehouse-dev-default", "MessageAttributes": {"task_name": {"StringValue": "warehouse.search.tasks.reindex_project", "DataType": "String"}, "_datadog": {"DataType": "String", "StringValue": "{\\"x-datadog-trace-id\\": \\"185612277372681671\\", \\"x-datadog-parent-id\\": \\"4452118822582909169\\", \\"x-datadog-sampling-priority\\": \\"1\\", \\"x-datadog-tags\\": \\"_dd.p.dm=-0,_dd.p.tid=66b3976a00000000\\", \\"traceparent\\": \\"00-66b3976a0000000002936d6202bffdc7-3dc91a7b15e278f1-01\\", \\"tracestate\\": \\"dd=p:3dc91a7b15e278f1;s:1;t.dm:-0;t.tid:66b3976a00000000\\"}"}}, "MessageBody": "eyJib2R5IjogIlcxc2lNbU5oY0hSamFHRWlYU3dnZTMwc0lIc2lZMkZzYkdKaFkydHpJam9nYm5Wc2JDd2dJbVZ5Y21KaFkydHpJam9nYm5Wc2JDd2dJbU5vWVdsdUlqb2diblZzYkN3Z0ltTm9iM0prSWpvZ2JuVnNiSDFkIiwgImNvbnRlbnQtZW5jb2RpbmciOiAidXRmLTgiLCAiY29udGVudC10eXBlIjogImFwcGxpY2F0aW9uL2pzb24iLCAiaGVhZGVycyI6IHsibGFuZyI6ICJweSIsICJ0YXNrIjogIndhcmVob3VzZS5zZWFyY2gudGFza3MucmVpbmRleF9wcm9qZWN0IiwgImlkIjogIjgzYzMxNmE4LWViMjQtNDEyNC1hYzkwLWQzNDE0ZDAzMDViNyIsICJzaGFkb3ciOiBudWxsLCAiZXRhIjogbnVsbCwgImV4cGlyZXMiOiBudWxsLCAiZ3JvdXAiOiBudWxsLCAiZ3JvdXBfaW5kZXgiOiBudWxsLCAicmV0cmllcyI6IDAsICJ0aW1lbGltaXQiOiBbbnVsbCwgbnVsbF0sICJyb290X2lkIjogIjgzYzMxNmE4LWViMjQtNDEyNC1hYzkwLWQzNDE0ZDAzMDViNyIsICJwYXJlbnRfaWQiOiBudWxsLCAiYXJnc3JlcHIiOiAiKCcyY2FwdGNoYScsKSIsICJrd2FyZ3NyZXByIjogInt9IiwgIm9yaWdpbiI6ICJnZW44QDJiNzU2YzA3ODU1ZiIsICJpZ25vcmVfcmVzdWx0IjogdHJ1ZSwgInJlcGxhY2VkX3Rhc2tfbmVzdGluZyI6IDAsICJzdGFtcGVkX2hlYWRlcnMiOiBudWxsLCAic3RhbXBzIjoge30sICJzZW50cnktdGFzay1lbnF1ZXVlZC10aW1lIjogMTcyMzA0NTczOC4xNTUzOTAzLCAic2VudHJ5LXRyYWNlIjogIjM4N2QwZWY0YjllMTQyNDM4YmU3YWJlMGJhNjBkNzlmLWE4YzI4OTUwMjEwYTBiZWIiLCAiYmFnZ2FnZSI6ICJzZW50cnktdHJhY2VfaWQ9Mzg3ZDBlZjRiOWUxNDI0MzhiZTdhYmUwYmE2MGQ3OWYsc2VudHJ5LWVudmlyb25tZW50PXByb2R1Y3Rpb24sc2VudHJ5LXJlbGVhc2U9bnVsbCIsICJoZWFkZXJzIjogeyJzZW50cnktdHJhY2UiOiAiMzg3ZDBlZjRiOWUxNDI0MzhiZTdhYmUwYmE2MGQ3OWYtYThjMjg5NTAyMTBhMGJlYiIsICJiYWdnYWdlIjogInNlbnRyeS10cmFjZV9pZD0zODdkMGVmNGI5ZTE0MjQzOGJlN2FiZTBiYTYwZDc5ZixzZW50cnktZW52aXJvbm1lbnQ9cHJvZHVjdGlvbixzZW50cnktcmVsZWFzZT1udWxsIiwgInNlbnRyeS10YXNrLWVucXVldWVkLXRpbWUiOiAxNzIzMDQ1NzM4LjE1NTM5MDN9fSwgInByb3BlcnRpZXMiOiB7ImNvcnJlbGF0aW9uX2lkIjogIjgzYzMxNmE4LWViMjQtNDEyNC1hYzkwLWQzNDE0ZDAzMDViNyIsICJyZXBseV90byI6ICIyYjY1ODY5MS1jN2JjLTNjMmEtYjM5NS05ZTY3Y2QxNDVkODgiLCAiZGVsaXZlcnlfbW9kZSI6IDIsICJkZWxpdmVyeV9pbmZvIjogeyJleGNoYW5nZSI6ICIiLCAicm91dGluZ19rZXkiOiAiZGVmYXVsdCJ9LCAicHJpb3JpdHkiOiAwLCAiYm9keV9lbmNvZGluZyI6ICJiYXNlNjQiLCAiZGVsaXZlcnlfdGFnIjogIjQ4NDM1NWE5LTYwMTYtNGY0ZS1hZjc2LTI3ZDBlYjZlOTU4ZiJ9fQ=="}', 'url': 'http://localstack:4566/', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0xffff83d31880>, 'has_streaming_input': False, 'auth_type': None}}", "thread": 281473237798944}

We can see that both task_name and _datadog keys are added to the MessageAttributes dictionary - so the mechanics to emit them is correct.

What is the result that you expected?

Spans for aws.sqs.sendmessage to have an extra attribute named task_name with the associated string value and can be viewed/searched/excluded for in the Datadog Trace UI.