Zipkin tags object disabled in ES

[vineetalagh]

Describe the Bug

Zipkin version: 2.23.2

ES Version: AWS ES 7.9

I have setup zipkin to write spans to our AWS managed ES instance. I am unable to search tags object in ES because they are not searchable. They have been marked as unknown datatype and are not enabled in the template.

How can I resolve this?

The zipkin template is:

{
  "zipkin-span-2021-08-11" : {
    "aliases" : { },
    "mappings" : {
      "_source" : {
        "excludes" : [ "_q" ]
      },
      "dynamic_templates" : [ {
        "strings" : {
          "match" : "*",
          "match_mapping_type" : "string",
          "mapping" : {
            "ignore_above" : 256,
            "norms" : false,
            "type" : "keyword"
          }
        }
      } ],
      "properties" : {
        "_q" : {
          "type" : "keyword"
        },
        "annotations" : {
          "type" : "object",
          "enabled" : false
        },
        "duration" : {
          "type" : "long"
        },
        "id" : {
          "type" : "keyword",
          "ignore_above" : 256
        },
        "kind" : {
          "type" : "keyword",
          "ignore_above" : 256
        },
        "localEndpoint" : {
          "dynamic" : "false",
          "properties" : {
            "serviceName" : {
              "type" : "keyword"
            }
          }
        },
        "name" : {
          "type" : "keyword"
        },
        "parentId" : {
          "type" : "keyword",
          "ignore_above" : 256
        },
        "remoteEndpoint" : {
          "dynamic" : "false",
          "properties" : {
            "serviceName" : {
              "type" : "keyword"
            }
          }
        },
        "shared" : {
          "type" : "boolean"
        },
        "tags" : {
          "type" : "object",
          "enabled" : false
        },
        "timestamp" : {
          "type" : "long"
        },
        "timestamp_millis" : {
          "type" : "date",
          "format" : "epoch_millis"
        },
        "traceId" : {
          "type" : "keyword"
        }
      }
    },
    "settings" : {
      "index" : {
        "number_of_shards" : "5",
        "provided_name" : "zipkin-span-2021-08-11",
        "creation_date" : "1628640006251",
        "requests" : {
          "cache" : {
            "enable" : "true"
          }
        },
        "number_of_replicas" : "1",
        "uuid" : "-BYja4VdRb6PdKOd5nFxRg",
        "version" : {
          "created" : "7090199"
        }
      }
    }
  }
}

[jcchavezs]

Ping @xeraa

On Wed, 11 Aug 2021, 13:36 vineetalagh, @.***> wrote:

Describe the Bug

Zipkin version: 2.23.2

ES Version: AWS ES 7.9

I have setup zipkin to write spans to our AWS managed ES instance. I am unable to search tags object in ES because they are not searchable. They have been marked as unknown datatype and are not enabled in the template.

How can I resolve this?

The zipkin template is:

{ "zipkin-span-2021-08-11" : { "aliases" : { }, "mappings" : { "_source" : { "excludes" : [ "_q" ] }, "dynamic_templates" : [ { "strings" : { "match" : "*", "match_mapping_type" : "string", "mapping" : { "ignore_above" : 256, "norms" : false, "type" : "keyword" } } } ], "properties" : { "_q" : { "type" : "keyword" }, "annotations" : { "type" : "object", "enabled" : false }, "duration" : { "type" : "long" }, "id" : { "type" : "keyword", "ignore_above" : 256 }, "kind" : { "type" : "keyword", "ignore_above" : 256 }, "localEndpoint" : { "dynamic" : "false", "properties" : { "serviceName" : { "type" : "keyword" } } }, "name" : { "type" : "keyword" }, "parentId" : { "type" : "keyword", "ignore_above" : 256 }, "remoteEndpoint" : { "dynamic" : "false", "properties" : { "serviceName" : { "type" : "keyword" } } }, "shared" : { "type" : "boolean" }, "tags" : { "type" : "object", "enabled" : false }, "timestamp" : { "type" : "long" }, "timestamp_millis" : { "type" : "date", "format" : "epoch_millis" }, "traceId" : { "type" : "keyword" } } }, "settings" : { "index" : { "number_of_shards" : "5", "provided_name" : "zipkin-span-2021-08-11", "creation_date" : "1628640006251", "requests" : { "cache" : { "enable" : "true" } }, "number_of_replicas" : "1", "uuid" : "-BYja4VdRb6PdKOd5nFxRg", "version" : { "created" : "7090199" } } } } }

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/openzipkin/zipkin/issues/3366, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXOYATPU4I64N7GB2ORPA3T4JOEBANCNFSM5B6IJYSA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email .

[xeraa]

Looking at the relevant part of the mapping:

"tags" : {
    "type" : "object",
    "enabled" : false
}

I'm afraid "enabled" : false is exactly doing that:

The JSON can still be retrieved from the _source field, but it is not searchable or stored in any other way

Besides reducing the storage overhead this would also allow you to have different types of data under that field, since Elasticsearch will skip parsing the field. Maybe this should be a keyword as well (which also covers an array or keywords)?

PS: You cannot change this setting in an existing index. You'll need to adjust the template for tomorrow (and you could reindex existing data, but that might be an overkill).

[Layfolk-zcy]

Looking at the relevant part of the mapping:

"tags" : {
    "type" : "object",
    "enabled" : false
}

I'm afraid "enabled" : false is exactly doing that:

The JSON can still be retrieved from the _source field, but it is not searchable or stored in any other way

Besides reducing the storage overhead this would also allow you to have different types of data under that field, since Elasticsearch will skip parsing the field. Maybe this should be a keyword as well (which also covers an array or keywords)?

PS: You cannot change this setting in an existing index. You'll need to adjust the template for tomorrow (and you could reindex existing data, but that might be an overkill).

The same problem, rebuilding index mapping is only a temporary solution, how to solve this problem from the root?

[xeraa]

The general fix is to change it in the template, so all future indices will use the new mapping.

With Elasticsearch 7.11+ you also have runtime fields, which would allow you to extract information from _source. Fixing bad mappings (at the cost of higher query time overhead) is one of the use-cases for runtime fields.