apm-agent-python

apm-agent-python copied to clipboard

Is your feature request related to a problem? Please describe.

When opening an issue with AWS Support, they will ask for the Request ID. If you don't have this ID then they won't help you. You are out of luck if you can't reproduce the issue.

Describe the solution you'd like I would like to add the RequestID somewhere on the span for the boto3 calls. I couldn't find somewhere obvious to put it in the Elastic Common Schema (it doesn't matter to me where its stored on the span). I am mostly interested in getting this ID for errors but it seems simple enough to add it for errors and successful calls. Something like this (from your botocore.py):

from botocore.exceptions import ClientError

def call(self, module, method, wrapped, instance, args, kwargs):
      ...
      with capture_span(
            handler_info.signature,
            span_type=handler_info.span_type,
            leaf=True,
            span_subtype=handler_info.span_subtype,
            span_action=handler_info.span_action,
            extra=handler_info.context,
        ) as span:
            if service in span_modifiers:
                span_modifiers[service](span, args, kwargs)
            try:
                response = wrapped(*args, **kwargs)
                add_response_meta_data(span, response)
                return response
            except ClientError as client_error:
                add_response_meta_data(span, client_error.response)
                raise
            


def add_response_meta_data(span, response):
    if response:
        request_id = response.get('ResponseMetadata', {}).get('RequestId')
        span.label(RequestId = request_id)

I can submit a PR and tests if that helps.

Describe alternatives you've considered Its possible to get the request ID if you add very verbose logging (as described here: https://github.com/boto/boto3/issues/1143) but that creates a lot of noise in the logs.

Hmmm, it seems like a no-brainer that we should be collecting this information. For our lambda integration, we do collect the request ID and store it in faas.execution and faas.trigger.request_id. But I don't think we should be doing anything faas-related in non-lambda transactions/spans. I'm going to ask around if other agents think we should get this added to intake -- if not, we can use labels as you've proposed.

I opened https://github.com/elastic/apm-server/issues/7871 to discuss adding span.context.http.request.id to the intake, which I think would be a great location for this Request ID. We could also potentially stick it in span.context.response.headers which is already supported in the intake. Stay tuned.

Thanks I really appreciate it!