jaegertracing
[Bug]: Data type mismatch between Jaeger v1 and v2 when using Elasticsearch as backend
What happened?
There is a difference in how int64 tag values are stored when using Elasticsearch as the span backend between Jaeger v1 and v2.
use the below DSL to query jaeger v1 and jaeger v2 index:
{
"query": {
"match_all": {}
},
"size": 1
}
The jaeger v2 tag data for int64 type:
"tags" : [
{
"key" : "db.rows_affected",
"type" : "int64",
"value" : 1
}
]
The jaeger v1 tag data for int64 type:
"tags" : [
{
"key" : "http.status_code",
"type" : "int64",
"value" : "200"
}
]
Steps to reproduce
- jaeger v1 use version : 1.65.0
- jaeger v2 use version: 2.6.0
Expected behavior
It is unclear whether jaeger-v2 is intended to maintain the same data structure as jaeger-v1 in Elasticsearch. Nevertheless, this subtle discrepancy may go unnoticed and could introduce challenges for users migrating from jaeger-v1 to jaeger-v2.
Relevant log output
Screenshot
screemshot from kibana
jaeger v2:
jaeger v1:
Additional context
The complete jaeger v2 result:
{
"took" : 1180,
"timed_out" : false,
"_shards" : {
"total" : 3,
"successful" : 3,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 5337,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "jaeger-span-000001",
"_type" : "_doc",
"_id" : "GEmwNJcBsfy9iIP2sbex",
"_score" : 1.0,
"_source" : {
"traceID" : "2b4750bcdc225cfd9e908b876bb1b7f1",
"spanID" : "a41a4531488b3070",
"operationName" : "gorm.Query",
"references" : [
{
"refType" : "CHILD_OF",
"traceID" : "2b4750bcdc225cfd9e908b876bb1b7f1",
"spanID" : "db0baacba71114b5"
}
],
"startTime" : 1748935219942505,
"startTimeMillis" : 1748935219942,
"duration" : 4758,
"tags" : [
{
"key" : "otel.scope.name",
"type" : "string",
"value" : "github.com/uptrace/opentelemetry-go-extra/otelgorm"
},
{
"key" : "db.name",
"type" : "string",
"value" : "insight"
},
{
"key" : "server.address",
"type" : "string",
"value" : "mcamel-common-mysql-cluster-mysql-master.mcamel-system.svc.cluster.local:3306"
},
{
"key" : "db.system",
"type" : "string",
"value" : "mysql"
},
{
"key" : "db.statement",
"type" : "string",
"value" : "SELECT * FROM `rules` WHERE id = '01jvkj9g3qb8z083kdvtfnhgrp' ORDER BY `rules`.`id` LIMIT 1"
},
{
"key" : "db.sql.table",
"type" : "string",
"value" : "rules"
},
{
"key" : "db.rows_affected",
"type" : "int64",
"value" : 1
},
{
"key" : "span.kind",
"type" : "string",
"value" : "client"
}
],
"logs" : [ ],
"process" : {
"serviceName" : "insight-server",
"tags" : [
{
"key" : "k8s.node.name",
"type" : "string",
"value" : "cluster-bm-master1"
},
{
"key" : "k8s.pod.name",
"type" : "string",
"value" : "insight-server-58b64b85c-hvmqx"
},
{
"key" : "service.version",
"type" : "string",
"value" : "v0.36.1"
},
{
"key" : "k8s.cluster.name",
"type" : "string",
"value" : "kpanda-global-cluster"
}
],
"tag" : {
"k8s@cluster@id" : "a32bf78c-48e6-46a0-9618-a00c235de31e",
"k8s@namespace@name" : "insight-system"
}
}
}
}
]
}
}
The complete jaeger v1 result:
{
"took" : 994,
"timed_out" : false,
"_shards" : {
"total" : 3,
"successful" : 3,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 10000,
"relation" : "gte"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "jaeger-span-000001",
"_type" : "_doc",
"_id" : "dkmyNJcBsfy9iIP2jcwO",
"_score" : 1.0,
"_source" : {
"traceID" : "852c027090609665660d277db284707b",
"spanID" : "f86e16285af0f57a",
"operationName" : "/healthz",
"references" : [ ],
"startTime" : 1748933186542688,
"startTimeMillis" : 1748933186542,
"duration" : 16,
"tags" : [
{
"key" : "otel.scope.name",
"type" : "string",
"value" : "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
},
{
"key" : "otel.scope.version",
"type" : "string",
"value" : "semver:0.36.3"
},
{
"key" : "net.transport",
"type" : "string",
"value" : "ip_tcp"
},
{
"key" : "net.peer.ip",
"type" : "string",
"value" : "127.0.0.6"
},
{
"key" : "net.peer.port",
"type" : "int64",
"value" : "51628"
},
{
"key" : "net.host.ip",
"type" : "string",
"value" : "10.233.70.152"
},
{
"key" : "net.host.port",
"type" : "int64",
"value" : "3010"
},
{
"key" : "http.target",
"type" : "string",
"value" : "/healthz"
},
{
"key" : "http.server_name",
"type" : "string",
"value" : "ghippo-audit"
},
{
"key" : "http.route",
"type" : "string",
"value" : "/healthz"
},
{
"key" : "http.user_agent",
"type" : "string",
"value" : "kube-probe/1.30"
},
{
"key" : "http.scheme",
"type" : "string",
"value" : "http"
},
{
"key" : "http.host",
"type" : "string",
"value" : "10.233.70.152:3010"
},
{
"key" : "http.flavor",
"type" : "string",
"value" : "1.1"
},
{
"key" : "http.method",
"type" : "string",
"value" : "GET"
},
{
"key" : "http.status_code",
"type" : "int64",
"value" : "200"
},
{
"key" : "span.kind",
"type" : "string",
"value" : "server"
},
{
"key" : "internal.span.format",
"type" : "string",
"value" : "otlp"
}
],
"logs" : [ ],
"process" : {
"serviceName" : "ghippo-auditserver",
"tags" : [
{
"key" : "k8s.node.name",
"type" : "string",
"value" : "cluster-bm-master1"
},
{
"key" : "k8s.pod.name",
"type" : "string",
"value" : "ghippo-auditserver-59496bf87c-6r87t"
},
{
"key" : "k8s.cluster.name",
"type" : "string",
"value" : "kpanda-global-cluster"
}
],
"tag" : {
"k8s@cluster@id" : "a32bf78c-48e6-46a0-9618-a00c235de31e",
"k8s@namespace@name" : "ghippo-system"
}
}
}
}
]
}
}
Jaeger backend version
No response
SDK
No response
Pipeline
No response
Stogage backend
No response
Operating system
No response
Deployment model
No response
Deployment configs
Both your return value examples are labeled v2, which one is v1? If the v2 version returns the value as proper JSON number, the. I think it's the correct representation, even if different from v1.
What problem is this causing besides being different from jaeger-v1? I think storing numeric tags as numbers is the right behavior, as it allows aggregation queries over ES.
Hi @yurishkuro, thanks for the input! 🙌
I’ve submitted a PR that aligns the tag type handling and resolves the config/writer logic in the ES v2 setup (#7246). Let me know if anything needs adjustment or if further clarification is needed. Looking forward to your feedback!