opentelemetry-dotnet-contrib icon indicating copy to clipboard operation
opentelemetry-dotnet-contrib copied to clipboard

Published 20 hours ago • verified publisher icon open-telemetry

AWS OTel .NET SDK for Lambda doesn't create a parent span

Open tanaka-takayoshi opened this issue 2 years ago • 5 comments

When using AWS OTel .NET SDK with exmaple code, SDK didn't create parent span in descirbed here: https://github.com/open-telemetry/opentelemetry-dotnet-contrib/blob/main/src/OpenTelemetry.Contrib.Instrumentation.AWSLambda/Implementation/AWSLambdaWrapper.cs#L226

Due to this, HTTP Client span and AWS SDK clint span are created as a separated trace.

Here is a output when using Console Exporter.

2022-03-14T16:25:18.790+09:00 | 2022-03-14T07:25:18.790Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info Activity.TraceId: 49bfbc15d46972f5861736cc6e9c9aa1
-- | --
  | 2022-03-14T16:25:18.790+09:00 | 2022-03-14T07:25:18.790Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info Activity.SpanId: 368d99f0ab61c522
  | 2022-03-14T16:25:18.790+09:00 | 2022-03-14T07:25:18.790Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info Activity.TraceFlags: Recorded
  | 2022-03-14T16:25:18.790+09:00 | 2022-03-14T07:25:18.790Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info Activity.ActivitySourceName: OpenTelemetry.Instrumentation.Http
  | 2022-03-14T16:25:18.790+09:00 | 2022-03-14T07:25:18.790Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info Activity.DisplayName: HTTP GET
  | 2022-03-14T16:25:18.790+09:00 | 2022-03-14T07:25:18.790Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info Activity.Kind: Client
  | 2022-03-14T16:25:18.790+09:00 | 2022-03-14T07:25:18.790Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info Activity.StartTime: 2022-03-14T07:25:18.3324016Z
  | 2022-03-14T16:25:18.790+09:00 | 2022-03-14T07:25:18.790Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info Activity.Duration: 00:00:00.4578280
  | 2022-03-14T16:25:18.790+09:00 | 2022-03-14T07:25:18.790Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info Activity.Tags:
  | 2022-03-14T16:25:18.790+09:00 | 2022-03-14T07:25:18.790Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info http.method: GET
  | 2022-03-14T16:25:18.790+09:00 | 2022-03-14T07:25:18.790Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info http.host: checkip.amazonaws.com
  | 2022-03-14T16:25:18.790+09:00 | 2022-03-14T07:25:18.790Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info http.url: http://checkip.amazonaws.com/
  | 2022-03-14T16:25:18.790+09:00 | 2022-03-14T07:25:18.790Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info http.status_code: 200
  | 2022-03-14T16:25:18.790+09:00 | 2022-03-14T07:25:18.790Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info otel.status_code: UNSET
  | 2022-03-14T16:25:18.790+09:00 | 2022-03-14T07:25:18.790Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info Resource associated with Activity:
  | 2022-03-14T16:25:18.790+09:00 | 2022-03-14T07:25:18.790Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info cloud.provider: aws
  | 2022-03-14T16:25:18.790+09:00 | 2022-03-14T07:25:18.790Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info cloud.region: ap-northeast-1
  | 2022-03-14T16:25:18.790+09:00 | 2022-03-14T07:25:18.790Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info faas.name: HelloOtelInLambda-HelloWorldFunction-1G4CcTz3WvHy
  | 2022-03-14T16:25:18.791+09:00 | 2022-03-14T07:25:18.791Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info faas.version: $LATEST
  | 2022-03-14T16:25:18.791+09:00 | 2022-03-14T07:25:18.791Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info telemetry.sdk.name: opentelemetry
  | 2022-03-14T16:25:18.791+09:00 | 2022-03-14T07:25:18.791Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info telemetry.sdk.language: dotnet
  | 2022-03-14T16:25:18.791+09:00 | 2022-03-14T07:25:18.791Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info telemetry.sdk.version: 1.2.0.420
  | 2022-03-14T16:25:18.791+09:00 | 2022-03-14T07:25:18.791Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info service.name: HelloOtelInLambda-HelloWorldFunction-1G4CcTz3WvHy
  | 2022-03-14T16:25:18.791+09:00 | 2022-03-14T07:25:18.791Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info Activity.TraceId: fce44521978089ffaf917c5cbfb6d204
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info Activity.SpanId: 7f66cb3975dd76e1
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info Activity.TraceFlags: Recorded
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info Activity.ActivitySourceName: OpenTelemetry.Instrumentation.Http
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info Activity.DisplayName: HTTP POST
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info Activity.Kind: Client
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info Activity.StartTime: 2022-03-14T07:25:18.8346325Z
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info Activity.Duration: 00:00:00.0566086
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info Activity.Tags:
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info http.method: POST
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info http.host: 127.0.0.1:9001
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info http.url: http://127.0.0.1:9001/2018-06-01/runtime/invocation/6f6d5079-d304-4c32-9500-c41f32e13a4c/response
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info http.status_code: 202
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info otel.status_code: UNSET
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info Resource associated with Activity:
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info cloud.provider: aws
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info cloud.region: ap-northeast-1
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info faas.name: HelloOtelInLambda-HelloWorldFunction-1G4CcTz3WvHy
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info faas.version: $LATEST
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info telemetry.sdk.name: opentelemetry
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info telemetry.sdk.language: dotnet
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info telemetry.sdk.version: 1.2.0.420
  | 2022-03-14T16:25:18.891+09:00 | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info service.name: HelloOtelInLambda-HelloWorldFunction-1G4CcTz3WvHy
  | 2022-03-14T16:25:18.891+09:00コピー2022-03-14T07:25:18.891Z	6f6d5079-d304-4c32-9500-c41f32e13a4c	info | 2022-03-14T07:25:18.891Z 6f6d5079-d304-4c32-9500-c41f32e13a4c info

tanaka-takayoshi avatar Mar 14 '22 07:03 tanaka-takayoshi

Hi @tanaka-takayoshi Can you provide a simple repro code or a code snippet of your lambda function demonstrating the instrumentation? Also, what's the .Net runtime you're using for your function?

srprash avatar Mar 14 '22 17:03 srprash

After investigating my end, I finally found AWS X-Ray configuration is required when using AddAWSLambdaConfigurations method even if the trace is not sent to X-Ray but to other OTLP endpoint.

The lambda should have AWSXrayWriteOnlyAccess policy and Tracing should be Active. Is it intended behavior?

tanaka-takayoshi avatar Mar 16 '22 01:03 tanaka-takayoshi

@tanaka-takayoshi I am having the same problem. How did you manage to solve? What do you mean by AWS X-Ray configuration is required?

dglozano avatar Apr 26 '22 13:04 dglozano

@dglozano I enabled "Active tracing" by following the doc and added AWSXrayWriteOnlyAccess policy to a Lambda function. https://docs.aws.amazon.com/lambda/latest/dg/services-xray.html

According to this issue, AWSXrayWriteOnlyAccess policy might have been deprecated and AWSXRayDaemonWriteAccess should work. However, I confirmed with AWSXrayWriteOnlyAccess. I haven't tested with AWSXRayDaemonWriteAccess.

tanaka-takayoshi avatar Apr 26 '22 15:04 tanaka-takayoshi

We observed that when X-Ray is disabled, it will still pass an X-Ray trace ID to the Lambda but with the sampled flag not set. Thus, no Activity will be started by the instrumentation. Since 1.1.0-beta.2, there is an option AWSLambdaInstrumentationOptions.DisableAwsXRayContextExtraction that you can set during initialization that will disable parsing the X-Ray trace ID and thus should still create an activity without extra AWS/Lambda configuration. Does this solve your issue?

Oberon00 avatar Jul 17 '23 14:07 Oberon00

Stale issue. Closing based on @Oberon00 message. Please reopen if needed.

Kielek avatar Mar 05 '24 06:03 Kielek