aws-otel-lambda
aws-otel-lambda copied to clipboard
Published
20 hours ago •
aws-observability
aws-observability
OTel collector to report `Initialization` and `Overhead` segments of Lambda calls
Is your feature request related to a problem? Please describe.
At the moment, the only traces that get reported by the OTel collector are the ones that involve Lambda itself that belong to the Invocation segment of Lambda runtime. X-Ray supports reporting additional segments that Lambda runtime goes through: Initialization and Overhead and it can be valuable to report on these segments as well.
Describe the solution you'd like
OTel to report the full picture of Lambda lifecycle, including Overhead and Initialization, similar to how X-Ray reports it:
Hi @michaelhyatt ,
Thanks for the feature request.
The Lambda runtime will generate X-Ray segments (Initialization and Overhead) and export them to X-Ray backend using X-Ray Daemon. This process is integrated in Lambda and doesn't provide us any possibility to modify. As a result of that, the tracing SDKs are not able to be instrumented in the area that is outside the Lambda function handler (invocation). Hence, what we can do so far is to apply the trace header from the X-Ray segments, propagate it to the spans created by OTel SDK within invocation and further export them to X-Ray backend using OTel collector so that you're able to see a complete and connected trace graph. I would say changing the default X-Ray Daemon to OTel collector in Lambda would address your issue but currently we're really not confident to say that this will be done in near future. Any specific reason why you would like Initialization and Overhead segments to be exported by OTel collector?
Hi @lupengamzn
Thanks for getting back to me.
One such use case is to be able to track lambda cold starts and how long they take. I know I can get it through Lambda insights, but this requires ingesting a Cloudwatch logs feed which is not yet supported by ADOT and I would like to use ADOT to receive the data.
Another case is the spans reported by ADOT have parent.id that points at non-existing spans/segments. There are parent segments in X-Ray that represent Lambda runtime and API-GW, so I suspect these missing segments are the ones reported by either the Lambda runtime or API-GW.
This is the trace between 2 lambdas calling each other through the API-GW:
Timing and the 3 spans ADOT is reporting with unlinked (missing parents):
ADOT debug log with producer Span 0 being a child of Span 1, but Span 1 having a non-existent parent e4559134ec0f2a45
There are older events to load. Load more.
--
2021-06-29T00:28:51.050Z info builder/exporters_builder.go:97 Exporter started. { "kind": "exporter", "name": "awsxray" }
2021-06-29T00:28:51.050Z info builder/exporters_builder.go:92 Exporter is starting... { "kind": "exporter", "name": "logging" }
2021-06-29T00:28:51.050Z info builder/exporters_builder.go:97 Exporter started. { "kind": "exporter", "name": "logging" }
2021-06-29T00:28:51.050Z info service/service.go:205 Starting processors...
2021-06-29T00:28:51.050Z info builder/pipelines_builder.go:51 Pipeline is starting... { "pipeline_name": "traces", "pipeline_datatype": "traces" }
2021-06-29T00:28:51.050Z info builder/pipelines_builder.go:62 Pipeline is started. { "pipeline_name": "traces", "pipeline_datatype": "traces" }
2021-06-29T00:28:51.050Z info builder/pipelines_builder.go:51 Pipeline is starting... { "pipeline_name": "metrics", "pipeline_datatype": "metrics" }
2021-06-29T00:28:51.050Z info builder/pipelines_builder.go:62 Pipeline is started. { "pipeline_name": "metrics", "pipeline_datatype": "metrics" }
2021-06-29T00:28:51.050Z info service/service.go:210 Starting receivers...
2021-06-29T00:28:51.050Z info builder/receivers_builder.go:70 Receiver is starting... { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:51.050Z info otlpreceiver/otlp.go:83 Starting GRPC server on endpoint 0.0.0.0:4317 { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:51.050Z info otlpreceiver/otlp.go:145 Setting up a second GRPC listener on legacy endpoint 0.0.0.0:55680 { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:51.050Z info otlpreceiver/otlp.go:83 Starting GRPC server on endpoint 0.0.0.0:55680 { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:51.050Z info otlpreceiver/otlp.go:101 Starting HTTP server on endpoint 0.0.0.0:55681 { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:51.050Z info builder/receivers_builder.go:75 Receiver started. { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:51.050Z info service/application.go:197 Everything is ready. Begin running and processing data.
2021/06/29 00:28:51 Registered extension ID: "c292077b-dbed-4a4a-a787-18d28522fe9c"
2021/06/29 00:28:51 [collector] Register response: {
"functionName": "aws-python-api-worker-project-dev-producer",
"functionVersion": "$LATEST",
"handler": "handler.producer"
}
2021/06/29 00:28:51 [collector] Waiting for event...
START RequestId: c505bd1d-ccfa-435a-b013-1b6847f4abec Version: $LATEST
EXTENSION Name: cloudwatch_lambda_agent State: Ready Events: [INVOKE,SHUTDOWN]
EXTENSION Name: collector State: Ready Events: [INVOKE,SHUTDOWN]
2021/06/29 00:28:52 [collector] Received event: {
"eventType": "INVOKE",
"deadlineMs": 1624926538710,
"requestId": "c505bd1d-ccfa-435a-b013-1b6847f4abec",
"invokedFunctionArn": "arn:aws:lambda:ap-southeast-2:401722391821:function:aws-python-api-worker-project-dev-producer",
"tracing": {
"type": "X-Amzn-Trace-Id",
"value": "Root=1-60da6942-51511f142c4eb98a6271f0b0;Parent=14b8b85f0f6dcb45;Sampled=1"
}
}
2021/06/29 00:28:52 [collector] Waiting for event...
[INFO] 2021-06-29T00:28:52.714Z c505bd1d-ccfa-435a-b013-1b6847f4abec Message: {'resource': '/produce', 'path': '/produce', 'httpMethod': 'POST', 'headers': {'Accept': '*/*', 'CloudFront-Forwarded-Proto': 'https', 'CloudFront-Is-Desktop-Viewer': 'true', 'CloudFront-Is-Mobile-Viewer': 'false', 'CloudFront-Is-SmartTV-Viewer': 'false', 'CloudFront-Is-Tablet-Viewer': 'false', 'CloudFront-Viewer-Country': 'AU', 'content-type': 'application/json', 'Host': 'a76ahpomn5.execute-api.ap-southeast-2.amazonaws.com', 'User-Agent': 'curl/7.64.1', 'Via': '2.0 f64d112a663bbc2a59417646645d304f.cloudfront.net (CloudFront)', 'X-Amz-Cf-Id': '0tcPs_yXqo4W-pyPoHYmMIjEAJi1BIDRa0FpZluMhtNKAR3sE8K1UQ==', 'X-Amzn-Trace-Id': 'Root=1-60da6942-51511f142c4eb98a6271f0b0', 'X-Forwarded-For': '49.177.65.219, 130.176.108.67', 'X-Forwarded-Port': '443', 'X-Forwarded-Proto': 'https'}, 'multiValueHeaders': {'Accept': ['*/*'], 'CloudFront-Forwarded-Proto': ['https'], 'CloudFront-Is-Desktop-Viewer': ['true'], 'CloudFront-Is-Mobile-Viewer': ['false'], 'CloudFront-Is-SmartTV-Viewer': ['false'], 'CloudFront-Is-Tablet-Viewer': ['false'], 'CloudFront-Viewer-Country': ['AU'], 'content-type': ['application/json'], 'Host': ['a76ahpomn5.execute-api.ap-southeast-2.amazonaws.com'], 'User-Agent': ['curl/7.64.1'], 'Via': ['2.0 f64d112a663bbc2a59417646645d304f.cloudfront.net (CloudFront)'], 'X-Amz-Cf-Id': ['0tcPs_yXqo4W-pyPoHYmMIjEAJi1BIDRa0FpZluMhtNKAR3sE8K1UQ=='], 'X-Amzn-Trace-Id': ['Root=1-60da6942-51511f142c4eb98a6271f0b0'], 'X-Forwarded-For': ['49.177.65.219, 130.176.108.67'], 'X-Forwarded-Port': ['443'], 'X-Forwarded-Proto': ['https']}, 'queryStringParameters': None, 'multiValueQueryStringParameters': None, 'pathParameters': None, 'stageVariables': None, 'requestContext': {'resourceId': 'aiuted', 'resourcePath': '/produce', 'httpMethod': 'POST', 'extendedRequestId': 'BqViXEghywMFyrw=', 'requestTime': '29/Jun/2021:00:28:50 +0000', 'path': '/dev/produce', 'accountId': '401722391821', 'protocol': 'HTTP/1.1', 'stage': 'dev', 'domainPrefix': 'a76ahpomn5', 'requestTimeEpoch': 1624926530160, 'requestId': '55ae8ecf-b571-44e3-91eb-53713a069234', 'identity': {'cognitoIdentityPoolId': None, 'accountId': None, 'cognitoIdentityId': None, 'caller': None, 'sourceIp': '49.177.65.219', 'principalOrgId': None, 'accessKey': None, 'cognitoAuthenticationType': None, 'cognitoAuthenticationProvider': None, 'userArn': None, 'userAgent': 'curl/7.64.1', 'user': None}, 'domainName': 'a76ahpomn5.execute-api.ap-southeast-2.amazonaws.com', 'apiId': 'a76ahpomn5'}, 'body': '{"name": "John"}', 'isBase64Encoded': False}
[DEBUG] 2021-06-29T00:28:52.717Z c505bd1d-ccfa-435a-b013-1b6847f4abec Starting new HTTPS connection (1): a76ahpomn5.execute-api.ap-southeast-2.amazonaws.com:443
[DEBUG] 2021-06-29T00:28:55.225Z c505bd1d-ccfa-435a-b013-1b6847f4abec https://a76ahpomn5.execute-api.ap-southeast-2.amazonaws.com:443 "GET /dev/consume HTTP/1.1" 200 0
[DEBUG] 2021-06-29T00:28:55.226Z c505bd1d-ccfa-435a-b013-1b6847f4abec streaming subsegments...
[DEBUG] 2021-06-29T00:28:55.226Z c505bd1d-ccfa-435a-b013-1b6847f4abec sending: { "format": "json", "version": 1 }{ "id": "aa52e2561a104893", "name": "a76ahpomn5.execute-api.ap-southeast-2.amazonaws.com", "start_time": 1624926532.714969, "parent_id": "e4559134ec0f2a45", "in_progress": false, "http": { "request": { "method": "get", "url": "https://a76ahpomn5.execute-api.ap-southeast-2.amazonaws.com/dev/consume" }, "response": { "status": 200 } }, "trace_id": "1-60da6942-51511f142c4eb98a6271f0b0", "type": "subsegment", "namespace": "remote", "end_time": 1624926535.2263606 } to 169.254.79.2:2000.
2021-06-29T00:28:55.236Z INFO loggingexporter/logging_exporter.go:42 TracesExporter { "#spans": 2 }
2021-06-29T00:28:55.236Z DEBUG loggingexporter/logging_exporter.go:48 ResourceSpans #0
Resource labels:
-> telemetry.sdk.language: STRING(python)
-> telemetry.sdk.name: STRING(opentelemetry)
-> telemetry.sdk.version: STRING(1.2.0)
-> cloud.region: STRING(ap-southeast-2)
-> cloud.provider: STRING(aws)
-> faas.name: STRING(aws-python-api-worker-project-dev-producer)
-> faas.version: STRING($LATEST)
-> service.name: STRING(aws-python-api-worker-project-dev-producer)
InstrumentationLibrarySpans #0
InstrumentationLibrary opentelemetry.instrumentation.requests 0.21b0
Span #0
Trace ID : 60da694251511f142c4eb98a6271f0b0
Parent ID : 8b4abf4054e4a63e
ID : a05b87fbb86fafed
Name : HTTP GET
Kind : SPAN_KIND_CLIENT
Start time : 2021-06-29 00:28:52.715105133 +0000 UTC
End time : 2021-06-29 00:28:55.226321929 +0000 UTC
Status code : STATUS_CODE_UNSET
Status message :
Attributes:
-> http.method: STRING(GET)
-> http.url: STRING(https://a76ahpomn5.execute-api.ap-southeast-2.amazonaws.com/dev/consume)
-> http.status_code: INT(200)
Span #1
Trace ID : 60da694251511f142c4eb98a6271f0b0
Parent ID : e4559134ec0f2a45
ID : 8b4abf4054e4a63e
Name : handler.producer
Kind : SPAN_KIND_SERVER
Start time : 2021-06-29 00:28:52.71392873 +0000 UTC
End time : 2021-06-29 00:28:55.228214816 +0000 UTC
Status code : STATUS_CODE_UNSET
Status message :
Attributes:
-> faas.execution: STRING(c505bd1d-ccfa-435a-b013-1b6847f4abec)
-> faas.id: STRING(arn:aws:lambda:ap-southeast-2:401722391821:function:aws-python-api-worker-project-dev-producer)
-> faas.name: STRING(aws-python-api-worker-project-dev-producer)
-> faas.version: STRING($LATEST)
END RequestId: c505bd1d-ccfa-435a-b013-1b6847f4abec
REPORT RequestId: c505bd1d-ccfa-435a-b013-1b6847f4abec Duration: 2567.60 ms Billed Duration: 2568 ms Memory Size: 1024 MB Max Memory Used: 176 MB Init Duration: 1978.96 ms XRAY TraceId: 1-60da6942-51511f142c4eb98a6271f0b0 SegmentId: 14b8b85f0f6dcb45 Sampled: true
2021/06/29 00:34:50 [collector] Received event: {
"eventType": "SHUTDOWN",
"deadlineMs": 1624926892349,
"requestId": "",
"invokedFunctionArn": "",
"tracing": {
"type": "",
"value": ""
}
}
2021-06-29T00:34:50.950Z info service/application.go:212 Received stop test request
2021-06-29T00:34:50.950Z info service/application.go:307 Starting shutdown...
2021-06-29T00:34:50.950Z info service/service.go:225 Stopping receivers...
I also expected the consumer span 0 to be connected to the producer span 0 referring it as it's parent, but it is referring to yet another non-existent parent span.
Consumer ADOT debug log
No older events at this moment. Retry
--
START RequestId: f11c9820-cf23-4805-9ab2-a9b7dbb74ecc Version: $LATEST
LOGS Name: cloudwatch_lambda_agent State: Subscribed Types: [platform]
2021/06/29 00:28:53 [collector] Launching OpenTelemetry Lambda extension, version: v0.1.0
2021-06-29T00:28:53.487Z info service/application.go:277 Starting otelcol... { "Version": "v0.1.0", "NumCPU": 2 }
2021-06-29T00:28:53.487Z info service/application.go:185 Setting up own telemetry...
2021-06-29T00:28:53.487Z info service/application.go:220 Loading configuration...
2021-06-29T00:28:53.490Z info service/application.go:236 Applying configuration...
2021-06-29T00:28:53.493Z info builder/exporters_builder.go:274 Exporter was built. { "kind": "exporter", "exporter": "awsxray" }
2021-06-29T00:28:53.494Z info builder/exporters_builder.go:274 Exporter was built. { "kind": "exporter", "exporter": "logging" }
2021-06-29T00:28:53.495Z info builder/exporters_builder.go:274 Exporter was built. { "kind": "exporter", "exporter": "otlp" }
2021-06-29T00:28:53.496Z info builder/pipelines_builder.go:204 Pipeline was built. { "pipeline_name": "traces", "pipeline_datatype": "traces" }
2021-06-29T00:28:53.496Z info builder/pipelines_builder.go:204 Pipeline was built. { "pipeline_name": "metrics", "pipeline_datatype": "metrics" }
2021-06-29T00:28:53.496Z info builder/receivers_builder.go:230 Receiver was built. { "kind": "receiver", "name": "otlp", "datatype": "traces" }
2021-06-29T00:28:53.496Z info builder/receivers_builder.go:230 Receiver was built. { "kind": "receiver", "name": "otlp", "datatype": "metrics" }
2021-06-29T00:28:53.496Z info service/service.go:155 Starting extensions...
2021-06-29T00:28:53.496Z info service/service.go:200 Starting exporters...
2021-06-29T00:28:53.496Z info builder/exporters_builder.go:92 Exporter is starting... { "kind": "exporter", "name": "awsxray" }
2021-06-29T00:28:53.496Z info builder/exporters_builder.go:97 Exporter started. { "kind": "exporter", "name": "awsxray" }
2021-06-29T00:28:53.496Z info builder/exporters_builder.go:92 Exporter is starting... { "kind": "exporter", "name": "logging" }
2021-06-29T00:28:53.496Z info builder/exporters_builder.go:97 Exporter started. { "kind": "exporter", "name": "logging" }
2021-06-29T00:28:53.496Z info builder/exporters_builder.go:92 Exporter is starting... { "kind": "exporter", "name": "otlp" }
2021-06-29T00:28:53.496Z info builder/exporters_builder.go:97 Exporter started. { "kind": "exporter", "name": "otlp" }
2021-06-29T00:28:53.496Z info service/service.go:205 Starting processors...
2021-06-29T00:28:53.496Z info builder/pipelines_builder.go:51 Pipeline is starting... { "pipeline_name": "traces", "pipeline_datatype": "traces" }
2021-06-29T00:28:53.496Z info builder/pipelines_builder.go:62 Pipeline is started. { "pipeline_name": "traces", "pipeline_datatype": "traces" }
2021-06-29T00:28:53.496Z info builder/pipelines_builder.go:51 Pipeline is starting... { "pipeline_name": "metrics", "pipeline_datatype": "metrics" }
2021-06-29T00:28:53.496Z info builder/pipelines_builder.go:62 Pipeline is started. { "pipeline_name": "metrics", "pipeline_datatype": "metrics" }
2021-06-29T00:28:53.496Z info service/service.go:210 Starting receivers...
2021-06-29T00:28:53.496Z info builder/receivers_builder.go:70 Receiver is starting... { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:53.496Z info otlpreceiver/otlp.go:83 Starting GRPC server on endpoint 0.0.0.0:4317 { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:53.496Z info otlpreceiver/otlp.go:145 Setting up a second GRPC listener on legacy endpoint 0.0.0.0:55680 { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:53.496Z info otlpreceiver/otlp.go:83 Starting GRPC server on endpoint 0.0.0.0:55680 { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:53.496Z info otlpreceiver/otlp.go:101 Starting HTTP server on endpoint 0.0.0.0:55681 { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:53.496Z info builder/receivers_builder.go:75 Receiver started. { "kind": "receiver", "name": "otlp" }
2021-06-29T00:28:53.496Z info service/application.go:197 Everything is ready. Begin running and processing data.
2021/06/29 00:28:53 Registered extension ID: "146f9fde-727c-4b6f-820e-d46c417b9c69"
2021/06/29 00:28:53 [collector] Register response: {
"functionName": "aws-python-api-worker-project-dev-consumer",
"functionVersion": "$LATEST",
"handler": "handler.consumer"
}
2021/06/29 00:28:53 [collector] Waiting for event...
EXTENSION Name: cloudwatch_lambda_agent State: Ready Events: [INVOKE,SHUTDOWN]
EXTENSION Name: collector State: Ready Events: [INVOKE,SHUTDOWN]
2021/06/29 00:28:55 [collector] Received event: {
"eventType": "INVOKE",
"deadlineMs": 1624926541168,
"requestId": "f11c9820-cf23-4805-9ab2-a9b7dbb74ecc",
"invokedFunctionArn": "arn:aws:lambda:ap-southeast-2:401722391821:function:aws-python-api-worker-project-dev-consumer",
"tracing": {
"type": "X-Amzn-Trace-Id",
"value": "Root=1-60da6942-51511f142c4eb98a6271f0b0;Parent=45e7cdc610535622;Sampled=1"
}
}
2021/06/29 00:28:55 [collector] Waiting for event...
[INFO] 2021-06-29T00:28:55.171Z f11c9820-cf23-4805-9ab2-a9b7dbb74ecc Message: {'resource': '/consume', 'path': '/consume', 'httpMethod': 'GET', 'headers': {'Accept': '*/*', 'Accept-Encoding': 'gzip, deflate', 'CloudFront-Forwarded-Proto': 'https', 'CloudFront-Is-Desktop-Viewer': 'true', 'CloudFront-Is-Mobile-Viewer': 'false', 'CloudFront-Is-SmartTV-Viewer': 'false', 'CloudFront-Is-Tablet-Viewer': 'false', 'CloudFront-Viewer-Country': 'AU', 'Content-Type': 'application/json', 'Host': 'a76ahpomn5.execute-api.ap-southeast-2.amazonaws.com', 'User-Agent': 'python-requests/2.25.1', 'Via': '1.1 56e387dd6408b297d863ce745cdb7d9e.cloudfront.net (CloudFront)', 'X-Amz-Cf-Id': 'O-uxGB_OVVkIjqUTAB4eM9iQaa1-2qv_Dy-0ab78nKao6waOKT3Bpg==', 'X-Amzn-Trace-Id': 'Self=1-60da6944-62cbb03f69fdc0de67dc8ea4;Root=1-60da6942-51511f142c4eb98a6271f0b0;Parent=aa52e2561a104893;Sampled=1', 'X-Forwarded-For': '13.211.71.78, 64.252.174.85', 'X-Forwarded-Port': '443', 'X-Forwarded-Proto': 'https'}, 'multiValueHeaders': {'Accept': ['*/*'], 'Accept-Encoding': ['gzip, deflate'], 'CloudFront-Forwarded-Proto': ['https'], 'CloudFront-Is-Desktop-Viewer': ['true'], 'CloudFront-Is-Mobile-Viewer': ['false'], 'CloudFront-Is-SmartTV-Viewer': ['false'], 'CloudFront-Is-Tablet-Viewer': ['false'], 'CloudFront-Viewer-Country': ['AU'], 'Content-Type': ['application/json'], 'Host': ['a76ahpomn5.execute-api.ap-southeast-2.amazonaws.com'], 'User-Agent': ['python-requests/2.25.1'], 'Via': ['1.1 56e387dd6408b297d863ce745cdb7d9e.cloudfront.net (CloudFront)'], 'X-Amz-Cf-Id': ['O-uxGB_OVVkIjqUTAB4eM9iQaa1-2qv_Dy-0ab78nKao6waOKT3Bpg=='], 'X-Amzn-Trace-Id': ['Self=1-60da6944-62cbb03f69fdc0de67dc8ea4;Root=1-60da6942-51511f142c4eb98a6271f0b0;Parent=aa52e2561a104893;Sampled=1'], 'X-Forwarded-For': ['13.211.71.78, 64.252.174.85'], 'X-Forwarded-Port': ['443'], 'X-Forwarded-Proto': ['https']}, 'queryStringParameters': None, 'multiValueQueryStringParameters': None, 'pathParameters': None, 'stageVariables': None, 'requestContext': {'resourceId': 's30841', 'resourcePath': '/consume', 'httpMethod': 'GET', 'extendedRequestId': 'BqVixGU7SwMFXSw=', 'requestTime': '29/Jun/2021:00:28:52 +0000', 'path': '/dev/consume', 'accountId': '401722391821', 'protocol': 'HTTP/1.1', 'stage': 'dev', 'domainPrefix': 'a76ahpomn5', 'requestTimeEpoch': 1624926532747, 'requestId': '1f9382a0-227b-499e-a286-1d95fdf25237', 'identity': {'cognitoIdentityPoolId': None, 'accountId': None, 'cognitoIdentityId': None, 'caller': None, 'sourceIp': '13.211.71.78', 'principalOrgId': None, 'accessKey': None, 'cognitoAuthenticationType': None, 'cognitoAuthenticationProvider': None, 'userArn': None, 'userAgent': 'python-requests/2.25.1', 'user': None}, 'domainName': 'a76ahpomn5.execute-api.ap-southeast-2.amazonaws.com', 'apiId': 'a76ahpomn5'}, 'body': None, 'isBase64Encoded': False}
2021-06-29T00:28:55.182Z INFO loggingexporter/logging_exporter.go:42 TracesExporter { "#spans": 1 }
2021-06-29T00:28:55.182Z DEBUG loggingexporter/logging_exporter.go:48 ResourceSpans #0
Resource labels:
-> telemetry.sdk.language: STRING(python)
-> telemetry.sdk.name: STRING(opentelemetry)
-> telemetry.sdk.version: STRING(1.2.0)
-> cloud.region: STRING(ap-southeast-2)
-> cloud.provider: STRING(aws)
-> faas.name: STRING(aws-python-api-worker-project-dev-consumer)
-> faas.version: STRING($LATEST)
-> service.name: STRING(aws-python-api-worker-project-dev-consumer)
InstrumentationLibrarySpans #0
InstrumentationLibrary opentelemetry.instrumentation.aws_lambda 0.16.dev0
Span #0
Trace ID : 60da694251511f142c4eb98a6271f0b0
Parent ID : f1f4a7d678442d55
ID : c632bd0f9ee111ce
Name : handler.consumer
Kind : SPAN_KIND_SERVER
Start time : 2021-06-29 00:28:55.171514198 +0000 UTC
End time : 2021-06-29 00:28:55.17204406 +0000 UTC
Status code : STATUS_CODE_UNSET
Status message :
Attributes:
-> faas.execution: STRING(f11c9820-cf23-4805-9ab2-a9b7dbb74ecc)
-> faas.id: STRING(arn:aws:lambda:ap-southeast-2:401722391821:function:aws-python-api-worker-project-dev-consumer)
-> faas.name: STRING(aws-python-api-worker-project-dev-consumer)
-> faas.version: STRING($LATEST)
END RequestId: f11c9820-cf23-4805-9ab2-a9b7dbb74ecc
REPORT RequestId: f11c9820-cf23-4805-9ab2-a9b7dbb74ecc Duration: 52.70 ms Billed Duration: 53 ms Memory Size: 1024 MB Max Memory Used: 169 MB Init Duration: 1997.45 ms XRAY TraceId: 1-60da6942-51511f142c4eb98a6271f0b0 SegmentId: 45e7cdc610535622 Sampled: true
2021/06/29 00:34:53 [collector] Received event: {
"eventType": "SHUTDOWN",
"deadlineMs": 1624926894919,
"requestId": "",
"invokedFunctionArn": "",
"tracing": {
"type": "",
"value": ""
}
}
2021-06-29T00:34:53.520Z info service/application.go:212 Received stop test request
2021-06-29T00:34:53.520Z info service/application.go:307 Starting shutdown...
2021-06-29T00:34:53.520Z info service/service.go:225 Stopping receivers...
For the record, leaving the Python code I used to create these two lambdas:
import logging
import os
import requests
from aws_xray_sdk.core import xray_recorder
from aws_xray_sdk.core import patch_all
patch_all()
logger = logging.getLogger()
logger.setLevel(logging.DEBUG)
URL = os.getenv('CONSUMER_API')
def producer(event, context):
logger.info(f'Message: {event}')
res = requests.get(url=URL, headers={
"Content-Type": "application/json"
})
return {'statusCode': res.status_code}
def consumer(event, context):
logger.info(f'Message: {event}')
return {'statusCode': 200}
@michaelhyatt Thanks for providing the context!
The parents for Lambda server spans do exist. They're exported by Daemon, not collector, and that's the reason why you're not seeing them in collector's log. You can open the raw trace context in X-Ray console to see that it should have a segment with Id(e4559134ec0f2a45) there, otherwise you'll not be able to see a complete and connected service graph like the one you provided at all. Can you confirm this on your end? Or does the service graph you provided fail to represent the actual workflow of your service?
Thanks, @lupengamzn. I can confirm that I can see the parent of the traces I am receiving from the OTel collector being the "Invocation" segments that are visible in X-Ray. Is there a way for these to be reported by the OTel collector as well?
My problem is not having these Invocation and API-GW segments reported by the OTel collector.
@michaelhyatt Thanks for the confirmation.
We can leave this thread open for more observability. At the same time, we'll wait to see if there is any update on Lambda side that we can adopt to get this addressed.
Tested the ADOT OTel collector traces in Jaeger and it is reporting missing parent traces.
@michaelhyatt, Thanks for bringing this up. We're aware of this situation, and currently using AWS OTel managed Lambda layer on both X-Ray and other tracing service is not fully supported. Sorry for the inconvenience and we'll try to include that in future updates.
This issue is stale because it has been open 90 days with no activity. If you want to keep this issue open, please just leave a comment below and auto-close will be canceled
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
This issue was closed because it has been marked as stale for 30 days with no activity.