dd-trace-go

dd-trace-go copied to clipboard

Hello there.

Recently I've found a case that breaks retry functionality of AWS SDK. If wrapped http client was used for AWS SDK, HTTP Instrumentation attempts to inject span context to http headers right after request signing. These injected headers were treated as non-signed header on first request and usually most requests are succeeded in first attempt. but if something went wrong (e.g. request timed out), aws sdk attempts retrying. If retrying is happen, request signing also performed again. but in this case, request signer recognizes previous injected trace propagation headers and treat these headers as part of signed header. and then, http instrumentation injects span context to http header carrier and this overwrites previous http header value - So calculated signature will be always mismatched and request retry will always fail with InvalidSignatureException.

Due to same reason, dd-trace-js skips span context injection if aws signature found (Also see https://github.com/DataDog/dd-trace-js/pull/205) - So it would be great if dd-trace-go has same behavior.

Hi @mooyoul can you clarify why you are using contrib/net/http instead of our aws integrations for tracing AWS calls? A workaround we are exploring is whether we can provide a generalized option that can specify which headers to skip propagation. One may already exist that can be leveraged here, so we can look into that.

Hello @katiehockman , Thanks for hitting me up!

AWS Instrumentation can only give you information about a single Operation call, not what's happening inside the Operation. Sometimes we can see latency spikes in DynamoDB Operations. We don't have visibility into whether the target query was actually slow to process on the DynamoDB backend, or whether it's due to the overhead of doing a DNS lookup, creating a new connection because the HTTP connection didn't exist, or whatever. To solve these problems, we use HTTP Instrumentation, and the information it provides gives us additional visibility into what's going on inside DynamoDB operation calls.

We also use this extended visibility to help us find the sweet spot when it comes to controlling the parameters for the HTTP Connection Pool used by the DynamoDB Client. Since our service uses DynamoDB "very heavily", tuning performance parameters is also a very important task for us (we have 20M MAUs, and normally use 10K+ DynamoDB Capacity Units).