[Feature Request] Add support for X-Ray on SAM Local

[hugoprudente]

Hi Team,

When trying to use the AWS SAM Local with Node.js per example it fails with the following error:

 Error: Missing AWS Lambda trace data for X-Ray. Expected _X_AMZN_TRACE_ID to be set.

This happens as the X-Ray Lambda already have the traces as per documentation: http://docs.aws.amazon.com/xray/latest/devguide/xray-services-lambda.html

So when using SAM Local this Trace Header is a missing.

[PaulMaddox]

Thanks for opening this.

I believe the steps we need to take to add this to SAM Local are:

1. Check if Tracing: Active is set in the SAM template for the function being invoked
2. If so, run the AWS X-Ray daemon as a sidecar container in invoke.go
3. Set the correct environment variables (e.g. _X_AMZN_TRACE_ID) in env.go

[svenemtell]

It would be nice to use X-Ray from lambdas run using AWS SAM Local BUT more important to me would be a way to silently discard X-Ray calls in the code. I want to be able to use the same code in production as when testing using AWS SAM Local and for that to work now I have written something like this (which I don't want to do):

if (process.env.AWS_SAM_LOCAL) { AWS = require('aws-sdk'); } else { // Only run AWS X-Ray when NOT running AWS SAM Local AWSXRay = require('aws-xray-sdk-core'); AWS = AWSXRay.captureAWS(require('aws-sdk')); }

[rvypandey]

+1

This will be a very valuable tool for us!

[Willis0826]

Could we using sam local with X-Ray now?

[metaskills]

It appears not yet. I've been using the process.env check above as mentioned till then. Not a big issue for me either way.

[jabalsad]

The primary use-case I have for enabling x-ray integration is really just so that I don't have to have different configuration in my local dev environment vs my production environment. I'd like to be able to have my code enable x-ray, and the SAM local simulation not dying or raising errors on me. I'm less opinionated about whether the requests actually get submitted to the x-ray service upstream, and care more about just being able to emulate it locally.

{"level":"info","ts":"2020-01-30T01:52:08.225Z","msg":"Starting MyService HTTP server..."}
START RequestId: a4e6c4c6-1e57-1961-65c4-d482d910f26a Version: $LATEST
{"level":"debug","ts":"2020-01-30T01:52:08.237Z","msg":"Created new Resource: {\"id\":\"4KML2o7ZZXR\",\"field\":\"111122223333\",\"name\":\"integrationtestconfig\",\"destinationArn\":\"arn:aws:s3:::some-bucket\"}"}
2020-01-30T01:52:08Z [Error] Suppressing AWS X-Ray context missing panic: failed to begin subsegment named 'dynamodb': segment cannot be found.
END RequestId: a4e6c4c6-1e57-1961-65c4-d482d910f26a
REPORT RequestId: a4e6c4c6-1e57-1961-65c4-d482d910f26a	Init Duration: 159.52 ms	Duration: 6.39 ms	Billed Duration: 100 ms	Memory Size: 128 MB	Max Memory Used: 27 MB
Invalid API Gateway Response Keys:
[snip snip]

[destinybonavita]

Any update on this?

[jfuss]

We are not looking to support X-Ray locally at this time.

Closing.

[hugoprudente]

Not planing to have the feature and closing it instead moving it to a backlog to see how the priority should grow it's not the best way to handle the issue.

You are simply saying here that the community feedback and customers using the aws-sam-cli doesn't matter.

[awsjeffg]

@hugoprudente thanks for the honest feedback, we appreciate it. I realize it is frustrating to have filed this issue 3 years ago and have it closed after all this time. We decided to close this issue as part of an effort on the team currently maintaining SAM cli to do a better job of communicating with community members about what our intentions are, even if the answer isn't the one you would prefer.

Having said that, let's talk about it. If sam cli were to "support" X-Ray, what would your ideal dev experience be? The original report indicates you just don't want the lambda to crash - is that it, or do you want sam cli to actually behave like X-Ray in the cloud?

[hugoprudente]

A first point is prevent the crash and prevent us dev to create unnecessary handling as the production will have X-Ray so the sam-cli that we use as dev/test environments also should handle it.

Supporting the feature is also necessary as what's the point to have something that I can't test my traces locally and only in production and if I need to test X-Ray locally I can't use SAM-CLI and do everything by hand so in the end it's better to do everything by hand as I have being doing instead wasting my time in the SAM-CLI as the effort of testing x-ray and adding and removing it to work with SAM-CLI doesn't make sense at all.

X-Ray already works locally publishing it to the X-RAY service or being intercepted locally by any UDP handlers so having the SDK on a similar fashion s the Lambda have so users can try simulate the best of the Lambda features locally and understand how do X-Ray will work for their use case it's just a extra plus.

From my use cases only if doesn't use X-Ray at all I have the SAM-CLI and if it has the probability of needing X-Ray in the feature I don't use it as the extra effort describing above of crating 2 lanes todo 1 thing I prefer to pay extra for the lambdas and test on the real environment than waste my developers time.

[qingchm]

@hugoprudente Hi there, first of all unfortunately the reason that we said that we're not looking to support it is 1) some components just cannot support x-ray locally, example: https://github.com/aws/aws-lambda-runtime-interface-emulator#level-of-support 2) local tracing with x-ray doesn't work if the x-ray resource is not created yet. We really want to support the idea of having local runs to capture more Lambda features but x-ray isn't one that we can support yet. However, we are more than willing to communicate with you to provide a workaround locally so that you don't have to waste your time modifying your templates. Here are 2 options that we will try to provide: 1) we can generate warnings instead of errors for tracing property so that the local run wouldn't fail 2) we can mock what x-ray tracing does by printing to stdout locally. Please give us a feedback on which workaround idea makes more sense to you so we can improve your experience using sam-cli!

[hugoprudente]

Hi @qingchm,

I didn't know which project was behind the creation of the AWS Lambda mock. The base support for this feature should be provided there it looks like.

Regarding the X-Ray Resource is not created, which resource exactly you are referring to? As AWS X-Ray only needs the AWS X-Ray SDK in the code that is provided by AWS Lambda and bootstrapped together with the code that executes the lambda_handler in the same context.

Regarding the options that you provided here are my observations:

1. Show the warning will continue to fail the code due to the missing embedded AWS X-Ray SDK in the lambda_handler, and if I add the SDK manually it will be different from the production, which will double the work of testing it locally.

2. Mock the Xray, it will only be possible if you embed the X-Ray SDK on the lambda_handler executer context before the executions, which will solve the issue.

---

Here is a little bit more context on how the AWS X-Ray works with AWS Lambda so we can trace the best approach to support minimal features on SAM CLI.

The AWS Lambda code that executes the lambda_handler when X-Ray is enabled somehow loads the AWS X-RAY SDK Core in python will be this:

from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all

xray_recorder.configure(service='AWS SAM Local')
patch_all()

Then when I execute my lambda_handler during the AWS Lambda execution the xray_recoreder will be on my context creating my segment.

The aws_xray_sdk.core will automatically try to send the information to UDP2000 port, which may or not listen or do something with it.

With a couple of lines of doc, It's easy on how to pick that information and sent it to AWS X-RAY Api or for a mock socket server for troubleshooting.

The most important bit here is how can we bootstrap all the language supporting X-Ray to enable the X-Ray Recorder with the basic configuration and particularities like Segment not mutable.

I'm not sure if this will be resolved here, or if it will be resolved on the AWS Lambda Runtime Interface Emulator.

Either way, I will be glad to help add a wrapper to the executor to it but I'm sure that the AWS Lambda Team would be able to provide more insights on how they are wrapping it so this could be consistent, with production/edge and provide the same limitations to be as "real" as possible, as I know that some particularities that only happens on X-Ray SDK with AWS Lambda.

Looking forward to hearing from you on which project will be responsible for the lambda_handler execution so I can help you with a simple example and we can plan the roadmap on how to add this feature in the upcoming future, as I'm sure I'm not the only one that put SAM-CLI a side due to this limitation.

[qingchm]

@hugoprudente Hi sorry for the confusion in my expressions, since I am studying on this issue just recently. But as for the first workaround, what I mean is to generate warning while skipping the requirement for the x-ray sdk. And by the creation what I mean is the daemon that the x-ray sdk communicates with. We need to integrate that into all environments as well in order for the sdk to work. Hopefully my words make better sense now. And to further address this issue, we are involving more discussions and I should be in touch with you soon!

[mgrandis]

Hi @hugoprudente, @jabalsad, @svenemtell: Are you still having issue running X-Ray instrumented code with the latest version of SAM CLI (1.15.0) and X-Ray SDK (npm 3.2.0)? If so, could you provide a minimal code sample and steps to reproduce the issue? I could not reproduce the issue (no error/crash) using:

• https://docs.aws.amazon.com/lambda/latest/dg/nodejs-tracing.html
• https://github.com/awsdocs/aws-lambda-developer-guide/blob/master/sample-apps/blank-nodejs/function/index.js

[aws-sam-cli-stale-bot]

This issue has not received a response in 14 days. If you want to keep this issue open, please leave a comment below and auto-close will be canceled.

[Dazza65]

I'm using 3.3.3 of the aws-sdk-xray nom package with 3.17.0 of the javascript sdk

Including the following line will generate the below error when invoking locally with v1.23.0 of the sam-cli. The lambda function works fine when deployed to AWS and captures the SSM traffic in Xray.

  const ssmClient = AWSXRay.captureAWSv3Client(new SSMClient({region: process.env.AWS_REGION} ));

START RequestId: e638f7ba-4740-4dc9-917e-5f019a7a6bd1 Version: $LATEST
2021-05-31T13:10:06.696Z        e638f7ba-4740-4dc9-917e-5f019a7a6bd1    INFO    Error: Missing AWS Lambda trace data fo    at Object.contextMissingRuntimeError [as contextMissing] (/var/task/node_modules/aws-xray-sdk-core/dist/lib/context    at async Runtime.exports.lambdaHandler [as handler] (/var/task/app.js:71:24)ddleware.js:6:22:71:25).js:81:49)
2021-05-31T13:10:06.700Z        e638f7ba-4740-4dc9-917e-5f019a7a6bd1    ERROR   Invoke Error    {"errorType":"Error","errorMessage":"Missing AWS Lambda trace data for X-Ray. Ensure Active Tracing is enabled and no subsegments are created outside the function handler.","stack":["Error: Missing AWS Lambda trace data for X-Ray. Ensure Active Tracing is enabled and no subsegments are created outside the function handler.","    at Object.contextMissingRuntimeError [as contextMissing] (/var/task/node_modules/aws-xray-sdk-core/dist/lib/context_utils.js:17:23)","    at Segment.resolveLambdaTraceData (/var/task/node_modules/aws-xray-sdk-core/dist/lib/env/aws_lambda.js:81:49)","    at Object.getSegment (/var/task/node_modules/aws-xray-sdk-core/dist/lib/context_utils.js:92:25)","    at Object.resolveSegment (/var/task/node_modules/aws-xray-sdk-core/dist/lib/context_utils.js:71:25)","    at /var/task/node_modules/aws-xray-sdk-core/dist/lib/patchers/aws3_p.js:63:67","    at /var/task/node_modules/@aws-sdk/middleware-content-length/dist/cjs/index.js:24:16","    at /var/task/node_modules/@aws-sdk/middleware-serde/dist/cjs/serializerMiddleware.js:6:12","    at async /var/task/node_modules/@aws-sdk/middleware-logger/dist/cjs/loggerMiddleware.js:6:22","    at async getData (/var/task/app.js:17:22)","    at async Runtime.exports.lambdaHandler [as handler] (/var/task/app.js:71:24)"]}
END RequestId: e638f7ba-4740-4dc9-917e-5f019a7a6bd1

[moretalk]

I can confirm the exact same error as @Dazza65 with

aws-sdk-xray-core 3.3.3 aws-sdk version 3.24.0 sam-cli v1.27.2

[seanlindo]

I'm also having this issue as well. I don't know how to use this feature without polluting my code.

[Rabadash8820]

Based on @qingchm's earlier answer:

... unfortunately the reason that we said that we're not looking to support it is 1) some components just cannot support x-ray locally, example: https://github.com/aws/aws-lambda-runtime-interface-emulator#level-of-support ...

So I just opened this issue over on the Runtime Interface Emulator repo to get things moving on their end. If everybody goes and rants on that repo, maybe this feature will actually be implemented in the RIE, and then SAM CLI can support it as well. 😛

[jfuss]

Is this still an issue with recent releases? I am using 1.61 locally and not seeing failures others are seeing here. I am using python but I could have a setup that doesn't trigger what others have seen. I am seeing warnings from the X-Ray Python SDK but my function still works locally (returns what I expected).

If someone tries this in recent versions, can you try to add a fake _X_AMZN_TRACE_ID to the Functions Env Vars or share a full example? The understanding from talking to teams is that should be the only thing we need to do. But I haven't been able to even fail the function locally when using X-Ray.

[valerena]

I can still see the issue with the a newer version of SAM CLI (1.73.0). To reproduce it, it's not enough to just configure X-Ray, but you need to actually make an AWS call with X-Ray configured.

The way I've reproduced this is to use the nodejs16.x "Serverless API" example, and change this line to get an X-Ray enabled DynamoDB client:

const xray = require('aws-xray-sdk-core');
const client = new dynamodb.DocumentClient();
const docClient = xray.captureAWSClient((client).service);

The other thing to take into account for the workaround for this error is that the fake _X_AMZN_TRACE_ID needs to have a specific format (otherwise, it will still fail, but with another error). The variable needs to be something like this: Root=1-63f3c410-4ea085a24d12dea053c79ff4;Parent=a0ba11b52fb8b69f;Sampled=1

[valerena]

Actually I just noticed that I was using a really old version of the X-Ray library aws-xray-sdk-core. And it looks like the issue doesn't exist anymore with newer versions of it. I was using 1.x, and the current version is 3.4.1. We might have to check if this is actually fixed in other runtimes too, but at least in NodeJS now the library continues silently if the X-Ray env var is not present (unlike the old behavior, where it failed).

[mbfreder]

I tried to reproduce the issue with Node16.x, but the error doesn't show up. I also tried with the java 11, then python 3.9 runtimes just to be very sure, but I was still able to invoke the functions without any errors. Based on that, I think we can safely close this issue

[hawflau]

Also confirmed it works in Go with aws-xray-sdk-go. I agree we can close this issue

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.