lambci
XRay _X_AMZN_TRACE_ID environment value change?
I am curious why in the commit below _X_AMZN_TRACE_ID=Root=1-dc99d00f-c079a84d433534434534ef0d;Parent=91ed514f1e5c03b2;Sampled=0 was changed to _X_AMZN_TRACE_ID='Parent=11560be54abce8ed'.
https://github.com/lambci/docker-lambda/commit/f65a259d7bc7fe46f97c81133d05b95b70d61328#diff-de2dfe904ac98a74495473e67a0d0e2cR19
This causes applications written with X-Ray to issue warnings to the log such as
2018-02-02T22:47:45,649Z WARN com.amazonaws.xray.contexts.LambdaSegmentContext ForkJoinPool.commonPool-worker-1 [10f4788e-4461-4e0f-8f62-94f76eae3a6c] [] [] _X_AMZN_TRACE_ID is missing a trace ID, parent ID, or sampling decision. Subsegment my-segment discarded.
When run in AWS these warnings do not appear. I am fairly confident the old format of the environment variable that includes a trace id and sampling decision is what AWS uses & libraries expect. This is documented here:
https://docs.aws.amazon.com/lambda/latest/dg/lambda-x-ray.html
Was there a reason for this change or is this a bug?
It's not a bug – if you look at the environment of the parent processes (ie, the processes that trigger your lambda functions), you'll see that that's the value of that environment variable – so that's what it is in the Dockerfile
When a function is executed, it assigns a particular _X_AMZN_TRACE_ID based on this parent environment variable – but only for the duration of the function invocation.
The correct way to tackle this would be to assign a new _X_AMZN_TRACE_ID each invocation in the mock logic – I might look at doing that – but you can override this yourself by setting that env variable when you run the docker container
Interesting. My use case is with AWS SAM Local. My first attempt was to do exactly this: override the environment variable myself, but at least so far with SAM Local the environment variable is not being overridden for some unknown reason. Seems strange as I am setting other environment variables via the same method without issue. Guessing there is some SAM local logic setting the environment variable back or failing to set it at all.
This works as expected (_X_AMZN_TRACE_ID is set to FOO):
docker run -e _X_AMZN_TRACE_ID=FOO --rm -v "$PWD":/var/task --entrypoint "env" lambci/lambda:java8
That would be cool if a randomized trace ID was set in the mock runtime. Since otherwise as you point out it won't be possible to change the trace ID if you invoke the lambda using the same container.
Yeah, I've been trying to find a balance between randomizing it and allowing users to override – but the problem is I can't just test for existence of the variable, because it will always exist. I might just test if it matches /^Parent=/ and only override it then – otherwise leave it as whatever the user specified it as (which would usually be something like /^Root=/)
That sounds reasonable like a reasonable plan to me. If you were really paranoid you could only override if the Parent matches the exact ID provided in the base image. If someone wants to set it to just Parent they could still do so with a custom ID.