_X_AMZN_TRACE_ID is not being propagated in CustomRuntimeEventLoop

[dragankoz]

As per the aws documentation, Custom runtimes

Processing tasks Get an event – Call the next invocation API to get the next event. The response body contains the event data. Response headers contain the request ID and other information.

Propagate the tracing header – Get the X-Ray tracing header from the Lambda-Runtime-Trace-Id header in the API response. Set the _X_AMZN_TRACE_ID environment variable locally with the same value. The X-Ray SDK uses this value to connect trace data between services.

Setting the TracingInterceptor in the dynamodb client, like this, doesn't allow xray to record the trace due to missing _X_AMZN_TRACE_ID environment variable.

       builder = builder
                .httpClient(UrlConnectionHttpClient.builder().build())
                .credentialsProvider(EnvironmentVariableCredentialsProvider.create())
                .region(Region.of(amazonAwsRegionString))
                .overrideConfiguration(ClientOverrideConfiguration.builder().addExecutionInterceptor(new TracingInterceptor()).build());

The logged error is, 2022-05-19 02:49:19.180 WARN 9 --- [pool-4-thread-1] c.a.xray.contexts.LambdaSegmentContext : _X_AMZN_TRACE_ID is missing a trace ID, parent ID, or sampling decision. Subsegment DynamoDb discarded.

Solution: In CustomRuntimeEventLoop.java, set the System property value "_X_AMZN_TRACE_ID"

Line 114:

                String xrayTraceId = response.getHeaders().getFirst("lambda-runtime-trace-id");
                if (StringUtils.hasText(xrayTraceId) && 
                         System.getenv().get("_X_AMZN_TRACE_ID") == null && 
                         System.getProperties().getProperty("_X_AMZN_TRACE_ID") == null) {
                                 System.getProperties().setProperty("_X_AMZN_TRACE_ID", xrayTraceId);
                }

[mhalbritter]

Is this a spring-native issue or does this happen on the JVM too?

[dragankoz]

Is this a spring-native issue or does this happen on the JVM too?

Only happens in spring-native, JVM is fine must be because of the different way its bootstrapped in the lambdas

[mhalbritter]

Thanks for that information. Do you happen to have a sample project with which I can play around?

[dragankoz]

https://github.com/dragankoz/serverless-springboot-dynamodb-example

[dragankoz]

In XRay console, the jvm version shows trace to DynamoDB but the Native version doesn't

In cloudwatch, native version shows, 2022-06-16 01:47:46.212 WARN 9 --- [pool-4-thread-1] c.a.xray.contexts.LambdaSegmentContext : _X_AMZN_TRACE_ID is missing a trace ID, parent ID, or sampling decision. Subsegment DynamoDb discarded.

[sdeleuze]

Spring Native is now superseded by Spring Boot 3 official native support, see the related reference documentation for more details.

As a consequence, I am closing this issue, and recommend trying your use case with latest Spring Boot 3 version. If you still experience the issue reported here, please open an issue directly on the related Spring project (Spring Framework, Data, Security, Boot, Cloud, etc.) with a reproducer.

Thanks for your contribution on the experimental Spring Native project, we hope you will enjoy the official native support introduced by Spring Boot 3.