aws-otel-collector
aws-otel-collector copied to clipboard
aws-observability
Aurora database incorrectly named in X-Ray Service Map when using opentelemetry-instrumentation-psycopg2
Describe the bug
I have the following situation:
- A microservice
payforadoptionwritten in Golang, uses https://github.com/aws/aws-xray-sdk-go/xray - A microserice
petadoptionshistorywritten in Python, uses https://github.com/open-telemetry/opentelemetry-python-contrib/tree/main/instrumentation/opentelemetry-instrumentation-psycopg2
Both services access the same adoptions Aurora database, but generate different identifiers. On the AWS X-Ray ServiceMap, the database therefore appears as two separate entities, instead of the same one.
I have checked with internal experts and it appears the Golang naming is correct and there’s an issue with how the Python data gets translated. Can you confirm that?
Steps to reproduce
This is how I am instrumenting the Python application, and specifically the requests to the DB:
# OTLP Tracing
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
# Exporter
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
# Propagation
from opentelemetry.propagate import set_global_textmap
from opentelemetry.propagators.aws import AwsXRayPropagator
# AWS X-Ray ID Generator
from opentelemetry.sdk.extension.aws.trace import AwsXRayIdGenerator
# Instrumentation
from opentelemetry.instrumentation.psycopg2 import Psycopg2Instrumentor
# Instrumentation
Psycopg2Instrumentor().instrument()
# ... main application code follows here
What did you expect to see?
There should be only one adoptions Aurora database in the X-Ray Service Map.
What did you see instead?
Two adoptions Aurora database entities in the X-Ray Service Map, even though they are the same.
Environment
requirements.txt:
opentelemetry-distro==0.32b0
opentelemetry-exporter-otlp==1.11.1
opentelemetry-exporter-otlp-proto-grpc==1.11.1
opentelemetry-exporter-otlp-proto-http==1.11.1
opentelemetry-instrumentation==0.32b0
opentelemetry-instrumentation-dbapi==0.32b0
opentelemetry-instrumentation-psycopg2==0.32b0
opentelemetry-instrumentation-wsgi==0.32b0
opentelemetry-propagator-aws-xray==1.0.1
opentelemetry-proto==1.11.1
opentelemetry-sdk==1.12.0rc2
opentelemetry-sdk-extension-aws==2.0.1
opentelemetry-semantic-conventions==0.32b0
opentelemetry-util-http==0.32b0
Additional context
A sample trace JSON of the DB call from X-Ray:
{
"Id": "1-63277b89-fe9f6799d89b4c7133e6878e",
"Duration": 0.151,
"LimitExceeded": false,
"Segments": [
{
"Id": "a012d5fc91c5c8b5",
"Document": {
"id": "a012d5fc91c5c8b5",
"name": "PetAdoptionsHistory",
"start_time": 1663531913.3896663,
"trace_id": "1-63277b89-fe9f6799d89b4c7133e6878e",
"end_time": 1663531913.5402176,
"fault": false,
"error": false,
"throttle": false,
"http": {
"request": {
"url": "http://11.0.190.90:8080/metrics",
"method": "GET",
"user_agent": "Prometheus/",
"client_ip": "11.0.190.90"
},
"response": {
"status": 200,
"content_length": 0
}
},
"aws": {
"xray": {
"auto_instrumentation": false
}
},
"metadata": {
"default": {
"http.flavor": "1.1",
"http.route": "/metrics",
"otel.resource.service.name": "PetAdoptionsHistory"
}
},
"subsegments": [
{
"id": "b5a0d77346b2a0f0",
"name": "adoptions",
"start_time": 1663531913.3907487,
"end_time": 1663531913.5392418,
"fault": false,
"error": false,
"throttle": false,
"aws": {
"xray": {
"auto_instrumentation": false
}
},
"sql": {
"url": "localhost/adoptions",
"sanitized_query": "SELECT count(*) FROM transactions_history",
"database_type": "postgresql",
"user": "postgres"
},
"namespace": "remote"
}
]
}
},
{
"Id": "3b2ef5b43ceff295",
"Document": {
"id": "3b2ef5b43ceff295",
"name": "adoptions",
"start_time": 1663531913.3907487,
"trace_id": "1-63277b89-fe9f6799d89b4c7133e6878e",
"end_time": 1663531913.5392418,
"parent_id": "b5a0d77346b2a0f0",
"inferred": true,
"sql": {
"url": "localhost/adoptions",
"sanitized_query": "SELECT count(*) FROM transactions_history",
"database_type": "postgresql",
"user": "postgres"
},
"origin": "Database::SQL"
}
}
]
}
Hi @lorchda
Thanks for providing the context. In your raw trace json, I do not even see the expected url for the adoptions db to be appended to the subsegment name. Can you provide the raw trace json for the trace where the payforadoption service is calling the adoptions db, so we can compare both subsegments and see which fields may be missing?
In the previous message, find the raw trace of the petadoptionhistory (Python) service.
Below, the raw trace of the payforadoption (Golang) service.
The two services don't call each other, but they access the same DB.
{
"Id": "1-63359902-6fe875c837ecff8d71f94b3d",
"Duration": 0.223,
"LimitExceeded": false,
"Segments": [
{
"Id": "4f52cbab47b10bf2",
"Document": {
"id": "4f52cbab47b10bf2",
"name": "Services-statusupdaterservicelambdafn37242E00-Ac59ZW6rPIaq",
"start_time": 1664456962.8809009,
"trace_id": "1-63359902-6fe875c837ecff8d71f94b3d",
"end_time": 1664456962.9268546,
"parent_id": "6615a0c6ee6c8f1b",
"aws": {
"account_id": "244622237421",
"function_arn": "arn:aws:lambda:eu-central-1:244622237421:function:Services-statusupdaterservicelambdafn37242E00-Ac59ZW6rPIaq",
"resource_names": [
"Services-statusupdaterservicelambdafn37242E00-Ac59ZW6rPIaq"
]
},
"origin": "AWS::Lambda::Function",
"subsegments": [
{
"id": "28aa0922d838b91f",
"name": "Invocation",
"start_time": 1664456962.8809538,
"end_time": 1664456962.9256988,
"aws": {
"function_arn": "arn:aws:lambda:eu-central-1:244622237421:function:Services-statusupdaterservicelambdafn37242E00-Ac59ZW6rPIaq"
},
"subsegments": [
{
"id": "af2e9a081678c231",
"name": "DynamoDB",
"start_time": 1664456962.882,
"end_time": 1664456962.926,
"http": {
"response": {
"status": 200
}
},
"aws": {
"retries": 0,
"region": "eu-central-1",
"operation": "UpdateItem",
"request_id": "07GKR0T8QD5G3JRS2JH72VK09JVV4KQNSO5AEMVJF66Q9ASUAAJG",
"table_name": "Services-ddbpetadoption7B7CFEC9-X8XMO56C97VD",
"resource_names": [
"Services-ddbpetadoption7B7CFEC9-X8XMO56C97VD"
]
},
"namespace": "aws"
}
]
},
{
"id": "81dfb6f6d5454023",
"name": "Overhead",
"start_time": 1664456962.9257793,
"end_time": 1664456962.926738,
"aws": {
"function_arn": "arn:aws:lambda:eu-central-1:244622237421:function:Services-statusupdaterservicelambdafn37242E00-Ac59ZW6rPIaq"
}
}
]
}
},
{
"Id": "b067c60766a382a2",
"Document": {
"id": "b067c60766a382a2",
"name": "PetSite",
"start_time": 1664456962.860372,
"trace_id": "1-63359902-6fe875c837ecff8d71f94b3d",
"end_time": 1664456963.083259,
"http": {
"request": {
"url": "http://servi-petsi-1fudsy3tpc4ct-900669318.eu-central-1.elb.amazonaws.com/Payment/MakePayment",
"method": "POST",
"client_ip": "3.69.114.15",
"x_forwarded_for": true
},
"response": {
"status": 200
}
},
"aws": {
"xray": {
"sampling_rule_name": "Default",
"sdk_version": "2.10.1",
"sdk": "X-Ray for .NET Core"
}
},
"metadata": {
"default": {
"PetId": "018",
"PetType": "kitten"
}
},
"service": {
"runtime": ".NET Core Framework",
"runtime_version": "6.0.7"
},
"subsegments": [
{
"id": "c661d665c0fb2a12",
"name": "Call Payment API",
"start_time": 1664456962.8625,
"end_time": 1664456963.062149,
"annotations": {
"PetId": "018",
"PetType": "kitten"
},
"subsegments": [
{
"id": "1b0f525a9d105107",
"name": "servi-payfo-i2qm58m76e7b-914273836.eu-central-1.elb.amazonaws.com",
"start_time": 1664456962.862719,
"end_time": 1664456963.062123,
"http": {
"request": {
"url": "http://servi-payfo-i2qm58m76e7b-914273836.eu-central-1.elb.amazonaws.com/api/home/completeadoption?petId=018&petType=kitten",
"method": "POST"
},
"response": {
"status": 200,
"content_length": 138
}
},
"namespace": "remote"
}
]
},
{
"id": "7fd7bf7577944290",
"name": "Post Message to SQS",
"start_time": 1664456963.062165,
"end_time": 1664456963.068725
},
{
"id": "7f7fd78f2a7a63c5",
"name": "Send Notification",
"start_time": 1664456963.068746,
"end_time": 1664456963.082837
}
]
}
},
{
"Id": "4ff6fd9c357099a1",
"Document": {
"id": "4ff6fd9c357099a1",
"name": "PetAdoptionStatusUpdater/prod",
"start_time": 1664456962.87,
"trace_id": "1-63359902-6fe875c837ecff8d71f94b3d",
"end_time": 1664456962.927,
"parent_id": "73cd5aac2386d72d",
"http": {
"request": {
"url": "https://qvju0zgr04.execute-api.eu-central-1.amazonaws.com/prod/",
"method": "PUT",
"user_agent": "Go-http-client/2.0",
"client_ip": "3.69.114.15",
"x_forwarded_for": true
},
"response": {
"status": 200,
"content_length": 0
}
},
"aws": {
"api_gateway": {
"account_id": "244622237421",
"rest_api_id": "qvju0zgr04",
"stage": "prod",
"request_id": "d61639a7-fb64-4911-a2ec-bf8e51de2ffd"
}
},
"annotations": {
"aws:api_id": "qvju0zgr04",
"aws:api_stage": "prod"
},
"metadata": {
"default": {
"extended_request_id": "ZOTYeGHOliAFk7g=",
"request_id": "d61639a7-fb64-4911-a2ec-bf8e51de2ffd"
}
},
"origin": "AWS::ApiGateway::Stage",
"resource_arn": "arn:aws:apigateway:eu-central-1::/restapis/qvju0zgr04/stages/prod",
"subsegments": [
{
"id": "3ae6edab2b98d0e0",
"name": "Lambda",
"start_time": 1664456962.872,
"end_time": 1664456962.9259999,
"http": {
"request": {
"url": "https://lambda.eu-central-1.amazonaws.com/2015-03-31/functions/arn:aws:lambda:eu-central-1:244622237421:function:Services-statusupdaterservicelambdafn37242E00-Ac59ZW6rPIaq/invocations",
"method": "ANY"
},
"response": {
"status": 200,
"content_length": 7
}
},
"aws": {
"function_name": "Services-statusupdaterservicelambdafn37242E00-Ac59ZW6rPIaq",
"region": "eu-central-1",
"operation": "Invoke",
"resource_names": [
"Services-statusupdaterservicelambdafn37242E00-Ac59ZW6rPIaq"
]
},
"namespace": "aws"
}
]
}
},
{
"Id": "6615a0c6ee6c8f1b",
"Document": {
"id": "6615a0c6ee6c8f1b",
"name": "Services-statusupdaterservicelambdafn37242E00-Ac59ZW6rPIaq",
"start_time": 1664456962.878,
"trace_id": "1-63359902-6fe875c837ecff8d71f94b3d",
"end_time": 1664456962.926,
"parent_id": "3ae6edab2b98d0e0",
"http": {
"response": {
"status": 200
}
},
"aws": {
"request_id": "e90fc1f6-db65-49dd-a7a3-bfd4458d2e83"
},
"origin": "AWS::Lambda",
"resource_arn": "arn:aws:lambda:eu-central-1:244622237421:function:Services-statusupdaterservicelambdafn37242E00-Ac59ZW6rPIaq"
}
},
{
"Id": "28ca1212e3084054",
"Document": {
"id": "28ca1212e3084054",
"name": "payforadoption",
"start_time": 1664456962.864093,
"trace_id": "1-63359902-6fe875c837ecff8d71f94b3d",
"end_time": 1664456963.0605092,
"parent_id": "1b0f525a9d105107",
"http": {
"request": {
"url": "http://servi-payfo-i2qm58m76e7b-914273836.eu-central-1.elb.amazonaws.com/api/home/completeadoption",
"method": "POST",
"client_ip": "3.69.114.15",
"x_forwarded_for": true
},
"response": {
"status": 200
}
},
"aws": {
"ecs": {
"container": "ip-11-0-222-12.eu-central-1.compute.internal"
},
"xray": {
"sdk_version": "1.7.0",
"sdk": "X-Ray for Go"
}
},
"annotations": {
"PetId": "018",
"PetType": "kitten"
},
"metadata": {
"default": {
"timeTakenSeconds": 0.196282438
}
},
"service": {
"runtime": "gc",
"runtime_version": "go1.15.15"
},
"origin": "AWS::ECS::Container",
"subsegments": [
{
"id": "7fd5e496425194b4",
"name": "adoptions@services-databaseb269d8bb-syhiyhh5ddex.cluster-c921pkk3ms9q.eu-central-1.rds.amazonaws.com",
"start_time": 1664456962.8642309,
"end_time": 1664456962.8691144,
"sql": {
"url": "postgres://postgres@services-databaseb269d8bb-syhiyhh5ddex.cluster-c921pkk3ms9q.eu-central-1.rds.amazonaws.com:5432/adoptions",
"sanitized_query": "\n\t\tINSERT INTO transactions (pet_id, transaction_id, adoption_date)\n\t\tVALUES ($1, $2, $3)\n\t",
"database_type": "Postgres",
"database_version": "PostgreSQL 10.18 on x86_64-pc-linux-gnu, compiled by x86_64-pc-linux-gnu-gcc (GCC) 7.4.0, 64-bit",
"driver_version": "github.com/lib/pq",
"user": "postgres"
},
"namespace": "remote"
},
{
"id": "0339204061b0acbb",
"name": "UpdateAvailability",
"start_time": 1664456962.8691344,
"end_time": 1664456963.0604098,
"subsegments": [
{
"id": "894a046f11e4fd06",
"name": "Update Adoption Status",
"start_time": 1664456962.8691604,
"end_time": 1664456962.9285643,
"subsegments": [
{
"id": "73cd5aac2386d72d",
"name": "qvju0zgr04.execute-api.eu-central-1.amazonaws.com",
"start_time": 1664456962.8692088,
"end_time": 1664456962.9285145,
"http": {
"request": {
"url": "https://qvju0zgr04.execute-api.eu-central-1.amazonaws.com/prod/",
"method": "PUT"
},
"response": {
"status": 200,
"content_length": 7
}
},
"namespace": "remote",
"subsegments": [
{
"id": "4e745c888b98e8dd",
"name": "request",
"start_time": 1664456962.8692565,
"end_time": 1664456962.869349
},
{
"id": "c305d053c74bfa4e",
"name": "response",
"start_time": 1664456962.8693564,
"end_time": 1664456962.928448
}
]
}
]
},
{
"id": "7c10b8f043133da7",
"name": "Invoking Availability API",
"start_time": 1664456962.869269,
"end_time": 1664456963.0604024,
"subsegments": [
{
"id": "513078692e3b26fe",
"name": "amazon.com",
"start_time": 1664456962.8692956,
"end_time": 1664456962.960827,
"http": {
"request": {
"url": "https://amazon.com",
"method": "GET"
},
"response": {
"status": 301,
"content_length": 163
}
},
"namespace": "remote",
"subsegments": [
{
"id": "837021744fea1362",
"name": "request",
"start_time": 1664456962.8693762,
"end_time": 1664456962.86942
},
{
"id": "d218f0a986d68008",
"name": "response",
"start_time": 1664456962.869428,
"end_time": 1664456962.9607928
}
]
},
{
"id": "02a2204f9ff6dedc",
"name": "www.amazon.com",
"start_time": 1664456962.9608762,
"end_time": 1664456963.0603995,
"http": {
"request": {
"url": "https://www.amazon.com/",
"method": "GET"
},
"response": {
"status": 200
}
},
"namespace": "remote",
"subsegments": [
{
"id": "c40ef2e46d2f9312",
"name": "request",
"start_time": 1664456962.9609025,
"end_time": 1664456962.960962
},
{
"id": "c139f6dd34be6964",
"name": "response",
"start_time": 1664456962.96097,
"end_time": 1664456963.0603325
}
]
}
]
}
]
}
]
}
},
{
"Id": "1acb018528ea52a7",
"Document": {
"id": "1acb018528ea52a7",
"name": "amazon.com",
"start_time": 1664456962.8692956,
"trace_id": "1-63359902-6fe875c837ecff8d71f94b3d",
"end_time": 1664456962.960827,
"parent_id": "513078692e3b26fe",
"inferred": true,
"http": {
"request": {
"url": "https://amazon.com",
"method": "GET"
},
"response": {
"status": 301,
"content_length": 163
}
}
}
},
{
"Id": "18c406c01f5f42ff",
"Document": {
"id": "18c406c01f5f42ff",
"name": "www.amazon.com",
"start_time": 1664456962.9608762,
"trace_id": "1-63359902-6fe875c837ecff8d71f94b3d",
"end_time": 1664456963.0603995,
"parent_id": "02a2204f9ff6dedc",
"inferred": true,
"http": {
"request": {
"url": "https://www.amazon.com/",
"method": "GET"
},
"response": {
"status": 200
}
}
}
},
{
"Id": "289976bb12be8070",
"Document": {
"id": "289976bb12be8070",
"name": "adoptions@services-databaseb269d8bb-syhiyhh5ddex.cluster-c921pkk3ms9q.eu-central-1.rds.amazonaws.com",
"start_time": 1664456962.8642309,
"trace_id": "1-63359902-6fe875c837ecff8d71f94b3d",
"end_time": 1664456962.8691144,
"parent_id": "7fd5e496425194b4",
"inferred": true,
"sql": {
"url": "postgres://postgres@services-databaseb269d8bb-syhiyhh5ddex.cluster-c921pkk3ms9q.eu-central-1.rds.amazonaws.com:5432/adoptions",
"sanitized_query": "\n\t\tINSERT INTO transactions (pet_id, transaction_id, adoption_date)\n\t\tVALUES ($1, $2, $3)\n\t",
"database_type": "Postgres",
"database_version": "PostgreSQL 10.18 on x86_64-pc-linux-gnu, compiled by x86_64-pc-linux-gnu-gcc (GCC) 7.4.0, 64-bit",
"driver_version": "github.com/lib/pq",
"user": "postgres"
},
"origin": "Database::SQL"
}
},
{
"Id": "3bc4c22538d62f29",
"Document": {
"id": "3bc4c22538d62f29",
"name": "DynamoDB",
"start_time": 1664456962.882,
"trace_id": "1-63359902-6fe875c837ecff8d71f94b3d",
"end_time": 1664456962.926,
"parent_id": "af2e9a081678c231",
"inferred": true,
"http": {
"response": {
"status": 200
}
},
"aws": {
"retries": 0,
"region": "eu-central-1",
"operation": "UpdateItem",
"request_id": "07GKR0T8QD5G3JRS2JH72VK09JVV4KQNSO5AEMVJF66Q9ASUAAJG",
"table_name": "Services-ddbpetadoption7B7CFEC9-X8XMO56C97VD",
"resource_names": [
"Services-ddbpetadoption7B7CFEC9-X8XMO56C97VD"
]
},
"origin": "AWS::DynamoDB::Table"
}
}
]
}
Huh, looks like it might be an instrumentation bug with not capturing the DB Connection string properly? @srprash can we look into the Python instrumentation to see how it captures attributes?
Here's the subsegment naming logic: https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/c5cd3a2bbe34fff6e6499052a214a38333856c0d/exporter/awsxrayexporter/internal/translator/segment.go#L148-L151
Also, @lorchda, is there any difference in the way you call the DB between the two applications? I just notice that in the Python app, the sql.url is captured as localhost/adoptions, whereas in Go it's postgres://postgres@services-databaseb269d8bb-syhiyhh5ddex.cluster-c921pkk3ms9q.eu-central-1.rds.amazonaws.com:5432/adoptions
The connection to the database is established as follows:
import psycopg2
conn_params = {
'database': 'adoptions',
'user': 'postgres'
'password': 'REDACTED',
'host': 'services-databaseb269d8bb-syhiyhh5ddex.cluster-c921pkk3ms9q.eu-central-1.rds.amazonaws.com'
}
db = psycopg2.connect(**conn_params)
sql = 'SELECT count(*) FROM transactions_history'
cur = db.cursor()
cur.execute(sql)
result = cur.fetchone()
db.commit()
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
can we look into the Python instrumentation to see how it captures attributes?
@willarmiros @lorchda
Seems like the Python instrumentation of db clients is not recording the connection string attribute which the adot collector uses to get the hostname from and name the db subsegment as dbname@hostname.
I checked this dbapi instrumentation and the entire repo to see if the DB_CONNECTION_STRING attribute was being set on a span anywhere but I did not find it.
@lorchda Can you confirm if the postgres span from your Python application has the db.connection_string attribute on it?
If is it not present, then you can open an issue on the otel python contrib repo regarding this attribute not being set.
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.
Can you confirm if the postgres span from your Python application has the db.connection_string attribute on it? If is it not present, then you can open an issue on the otel python contrib repo regarding this attribute not being set.
@lorchda Were you able to verify this? I can help open the issue upstream if you can provide the sample span.
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.
This issue was closed because it has been marked as stale for 30 days with no activity.