Elasticsearch 8.x support

[Tornhoof]

Requirement - what kind of business use case are you trying to solve?

Elastic recently released Elasticsearch 8.0, by now they've released 8.1.

Problem - what in Jaeger blocks you from solving the requirement?

The ES index can't be created anymore, I think the legacy template api part is now completely deprecated:

jaeger-collector    | {"level":"info","ts":1646824639.4679213,"caller":"config/config.go:205","msg":"Elasticsearch detected","version":8}
jaeger-collector    | {"level":"info","ts":1646824639.4697242,"caller":"zapgrpc/zapgrpc.go:129","msg":"Deprecation warning: 299 Elasticsearch-8.1.0-3700f7679f7d95e36da0b43762189bab189bc53a \"Legacy index templates are deprecated in favor of composable templates.\""}
jaeger-collector    | {"level":"fatal","ts":1646824639.4699457,"caller":"./main.go:84","msg":"Failed to create span writer","error":"elastic: Error 400 (Bad Request): unknown key [template] in the template  [type=parse_exception]","stacktrace":"main.main.func1\n\t./main.go:84\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/[email protected]/command.go:856\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/[email protected]/command.go:974\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/[email protected]/command.go:902\nmain.main\n\t./main.go:147\nruntime.main\n\truntime/proc.go:255"}

The https://www.elastic.co/guide/en/elasticsearch/reference/8.1/migrating-8.0.html lists now (somewhere in the middle)

The [create or update index template API](https://www.elastic.co/guide/en/elasticsearch/reference/8.1/indices-templates-v1.html)’s template parameter has been removed.

Details
In 6.0, we deprecated the template parameter in create or update index template requests in favor of using index_patterns. Support for the template parameter is now removed in 8.0.

Impact
Use the create or update index template API's index_patterns parameter. Requests that include the template parameter will return an error.

I think the code falls back to the unversioned mappings if it does not find a specific mapping

So, in my opinion the code should not fall back to the jaeger-span.json mapping but instead use the jaeger-span-7.json for newer versions, e.g. use the -7 jsons for 8 etc.

e.g. change line 42 from == 7 to >=7 https://github.com/jaegertracing/jaeger/blob/7872d1b07439c3f2d316065b1fd53e885b26a66f/plugin/storage/es/mappings/mapping.go#L40-L46

Workaround

Force the ES Version to 7 via --es.version=7

[yurishkuro]

Would you like to submit a PR? Besides the logic fix it would need to add v8 to the CI steps.

[Tornhoof]

Sure, I'll give it a try.

[Ashmita152]

I spent time investigating the effort in achieving it. I think this is going to be difficult now that we use same elasticsearch client olivere/elastic to support both of our elasticsearch and opensearch implementations.

Major challenge: Right now we are using v6 of olivere/elastic which doesn't have support for new index template. To support it we need to update to v7 of olivere/elastic specifically for https://github.com/olivere/elastic/pull/1458 and https://github.com/olivere/elastic/pull/1415 which inturn breaks the opensearch support.

Not sure how we should proceed.

cc @yurishkuro

[s7an-it]

Workaround

Force the ES Version to 7 via --es.version=7

I have tried latest helm operator installation and setting this parameter didn't result in anything better.

[bitfactory-sem-denbroeder]

Would it be possible for Jaeger to add a separate docker image/branch which maintains opensearch support and the main branch getting support for Elasticsearch 8.x ?

[smithyj]

hope to support elasticsearch 8.1.x

[pavolloffay]

We are working on bootstrapping Jaeger v2 (#3500) which will be based on OTEL collector. This will allow us to have ES exporter per ES version.

[DanielBruzualRW]

Is there any workaround currently available, e.g., creating the index manually?

I tried the OP's suggested workaround (with elastic 8.1.3) and I get the following error:

legacy template [jaeger-span] has index patterns [*jaeger-span-*] matching patterns from existing composable templates [.deprecation-indexing-template,.ml-anomalies-,.ml-state,.ml-stats,.monitoring-beats-mb,.monitoring-ent-search-mb,.monitoring-es-mb,.monitoring-kibana-mb,.monitoring-logstash-mb,.slm-history,.watch-history-16,ilm-history,logs,metrics,synthetics] with patterns (.deprecation-indexing-template => [.logs-deprecation.*],.ml-anomalies- => [.ml-anomalies-*],.ml-state => [.ml-state*],.ml-stats => [.ml-stats-*],.monitoring-beats-mb => [.monitoring-beats-8-*],.monitoring-ent-search-mb => [.monitoring-ent-search-8-*],.monitoring-es-mb => [.monitoring-es-8-*],.monitoring-kibana-mb => [.monitoring-kibana-8-*],.monitoring-logstash-mb => [.monitoring-logstash-8-*],.slm-history => [.slm-history-5*],.watch-history-16 => [.watcher-history-16*],ilm-history => [ilm-history-5*],logs => [logs-*-*],metrics => [metrics-*-*],synthetics => [synthetics-*-*]), use composable templates (/_index_template) instead

[kruczjak]

I would be thankful for any workaround too.

We need to upgrade our elasticsearch cluster but we are stuck because of this problem

[seilorjunior]

we also need this asap!!!!!!1

[foxracle]

we also need this asap! +1

[sergiomcalzada]

Hi,

Workaround

Force the ES Version to 7 via --es.version=7 and es.create-index-templates: false

Add add this two indexes. If some of them fails related to "index priority" just change it

PUT _index_template/jaeger-service
{
  "priority": 1,
  "template": {
    "settings": {
      "index": {
        "mapping": {
          "nested_fields": {
            "limit": "50"
          }
        },
        "requests": {
          "cache": {
            "enable": "true"
          }
        },
        "number_of_shards": "6",
        "number_of_replicas": "1"
      }
    },
    "mappings": {
      "_routing": {
        "required": false
      },
      "_size": {
        "enabled": false
      },
      "numeric_detection": false,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "dynamic": true,
      "dynamic_templates": [
        {
          "span_tags_map": {
            "path_match": "tag.*",
            "mapping": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        {
          "process_tags_map": {
            "path_match": "process.tag.*",
            "mapping": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        }
      ],
      "date_detection": true,
      "properties": {
        "operationName": {
          "type": "keyword"
        },
        "serviceName": {
          "type": "keyword"
        }
      }
    }
  },
  "index_patterns": [
    "*jaeger-service-*"
  ],
  "composed_of": []
}

PUT _index_template/jaeger-span
{
  "priority": 2,
  "template": {
    "settings": {
      "index": {
        "mapping": {
          "nested_fields": {
            "limit": "50"
          }
        },
        "requests": {
          "cache": {
            "enable": "true"
          }
        },
        "number_of_shards": "6",
        "number_of_replicas": "1"
      }
    },
    "mappings": {
      "_routing": {
        "required": false
      },
      "_size": {
        "enabled": false
      },
      "numeric_detection": false,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "dynamic": true,
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "dynamic_templates": [
        {
          "span_tags_map": {
            "path_match": "tag.*",
            "mapping": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        {
          "process_tags_map": {
            "path_match": "process.tag.*",
            "mapping": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        }
      ],
      "date_detection": true,
      "properties": {
        "traceID": {
          "type": "keyword"
        },
        "process": {
          "type": "object",
          "properties": {
            "tag": {
              "type": "object"
            },
            "serviceName": {
              "type": "keyword"
            },
            "tags": {
              "type": "nested",
              "properties": {
                "tagType": {
                  "type": "keyword"
                },
                "value": {
                  "type": "keyword"
                },
                "key": {
                  "type": "keyword"
                }
              }
            }
          }
        },
        "references": {
          "type": "nested",
          "properties": {
            "spanID": {
              "type": "keyword"
            },
            "traceID": {
              "type": "keyword"
            },
            "refType": {
              "type": "keyword"
            }
          }
        },
        "startTimeMillis": {
          "type": "date"
        },
        "flags": {
          "type": "integer"
        },
        "operationName": {
          "type": "keyword"
        },
        "parentSpanID": {
          "type": "keyword"
        },
        "tags": {
          "type": "nested",
          "properties": {
            "tagType": {
              "type": "keyword"
            },
            "value": {
              "type": "keyword"
            },
            "key": {
              "type": "keyword"
            }
          }
        },
        "duration": {
          "type": "long"
        },
        "spanID": {
          "type": "keyword"
        },
        "startTime": {
          "type": "long"
        },
        "tag": {
          "type": "object",
          "properties": {}
        },
        "logs": {
          "type": "nested",
          "properties": {
            "fields": {
              "type": "nested",
              "properties": {
                "tagType": {
                  "type": "keyword"
                },
                "value": {
                  "type": "keyword"
                },
                "key": {
                  "type": "keyword"
                }
              }
            },
            "timestamp": {
              "type": "long"
            }
          }
        }
      }
    }
  },
  "index_patterns": [
    "*jaeger-span-*"
  ],
  "composed_of": []
}

[aletor92]

Workaround

It Works ! thanks 👍

[Evesy]

As part of adding support for the newer index template API, it might be nice to have the option for Jaeger to create component templates instead of index templates.

With the current API we have a couple of other index templates targetting jaeger indices that all get merged together to perform the following modifications:

• Set custom default settings that apply to all our indices (Tier routing, refresh interval, translog settings etc.)
• Modify some field mappings (e.g. we set eager_global_ordinals on traceID for better search performance)

With the new index template API this won't be possible as only one index template wll be chosen (highest priority) and even if Jaeger supporting passing composed_of values to the template, which would allow us to pull in our default index settings, anything still explicitly set by Jaeger template (e.g. the traceID field mapping) would take precedence over our modifications.

I'm not sure if this is a common use case or not, otherwise we may choose to disable Jaeger index template creation and just managed them entirely ourselves, however pulling in any new field mappings etc. automatically when upgrading Jaeger is handy

[jasperjonker]

Thanks @sergiomcalzada that worked:

storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://elasticsearch.elastic-db:9200
        index-prefix: jaeger_
        version: 7                                  # Necessary as it doesn't work with 8 yet
        create-index-templates: false               # Necessary as it doesn't work with 8 yet

[sylvainOL]

Fyi, https://github.com/jaegertracing/spark-dependencies is not working with es8.x also.

That means it cannot really be used in production IMHO

[billalley]

We are experiencing problems connecting Jaeger to our existing ES setup due to this issue. I've gone ahead and followed the work around instructions above, currently using the Helm chart for setup.

Ex:

   cmdlineParams:
      es.version: 7
      es.create-index-templates: false 

But no matter I still see this error relating to the template index patterns:

"caller":"./main.go:69","msg":"Failed to create span writer","error":"elastic: Error 400 (Bad Request): legacy template [jaeger-span] has index patterns [*jaeger-span-*] matching patterns from existing composable templates

I have also tried removing the leading * from the templates, but the error message stayed the same which leads me to believe this is still on the Jaeger end of things.

Anyone have further workaround info/updates on ES8 support?

[billalley]

Looks like this was some yaml mangling - No matter what was specified, checking the config state on the ingester showed

spec:
  containers:
  - args:
    - --es.create-index-templates
    - --es.version=7

Solution: quote the false

   cmdlineParams:
      es.version: 7
      es.create-index-templates: "false"

Which becomes:

spec:
  containers:
  - args:
    - --es.create-index-templates=false
    - --es.version=7

[yurishkuro]

@Ashmita152 I came across https://github.com/olivere/elastic/issues/1533, where the author of the library unequivocally states:

Elastic v8 won't happen. I will polish v7 if I have the time and energy, but v8 won't happen. I'm sorry.

So there seems to be little point in upgrading to v7 of the library as it won't support v8 of ES.

Unlike our Cassandra implementation that uses an internal abstraction over gocql driver, the ES implementation has a large exposure to many different types from olivere/elastic driver. This complicates things a lot if we want to support both ESv8 and OpenSearch using different drivers.

ESv7 will be EOL-ed by 2023-08-01.

[ajax-osadchuk-r]

@yurishkuro Could you please clarify whether Jaeger is going to support ElasticSearch v8.+? At least, if there are any plans to do so in near times? This issue blocks our team and forces either to use outdated infrastructure or switch to another distributed tracing system.

[ajax-osadchuk-r]

@yurishkuro @pavolloffay Any thoughts about my previous comment?

[luanguimaraesla]

Thank you @sergiomcalzada. It works. We're also using ILM/aliases, which changes the configuration a little bit.

Script to apply all manifests

Make sure you named the files correctly

#! /bin/bash

function apply_ilm() {
	policy_name=$1
	file_name=$2
	curl -Ss -XPUT -H 'Content-Type: application/json' \
	    -d @"${file_name}" \
	    "http://localhost:9200/_ilm/policy/${policy_name}"
}

function apply_template() {
	template_name=$1
	file_name=$2
	curl -Ss -XPUT -H 'Content-Type: application/json' \
		-d @"${file_name}" \
		"http://localhost:9200/_index_template/${template_name}"
}

function apply_index() {
	index_name=$1
	file_name=$2
	curl -Ss -XPUT -H 'Content-Type: application/json' \
		-d @"${file_name}" \
		"http://localhost:9200/${index_name}"
}

apply_ilm "jaeger" jaeger.ilm.json
apply_template "jaeger-service" jaeger-service.tpl.json
apply_template "jaeger-span" jaeger-span.tpl.json
apply_index "jaeger-service-000001" jaeger-service-000001.idx.json
apply_index "jaeger-span-000001" jaeger-span-000001.idx.json

ILM manfest (jaeger.ilm.json)

{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_age": "4h"
          },
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "2d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Spans Composable Template (jaeger-span.tpl.json)

{
  "version": 3,
  "index_patterns": [
    "*jaeger-span-*"
  ],
  "priority": 101,
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "jaeger",
          "rollover_alias": "jaeger-span-write"
        },
        "mapping": {
          "nested_fields": {
            "limit": "50"
          }
        },
        "requests": {
          "cache": {
            "enable": "true"
          }
        },
        "number_of_shards": 5
      }
    },
    "aliases": {
      "jaeger-span-read": {}
    },
    "mappings": {
      "_routing": {
        "required": false
      },
      "numeric_detection": false,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "dynamic": true,
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "dynamic_templates": [
        {
          "span_tags_map": {
            "path_match": "tag.*",
            "mapping": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        {
          "process_tags_map": {
            "path_match": "process.tag.*",
            "mapping": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        }
      ],
      "date_detection": true,
      "properties": {
        "traceID": {
          "type": "keyword"
        },
        "process": {
          "type": "object",
          "properties": {
            "tag": {
              "type": "object"
            },
            "serviceName": {
              "type": "keyword"
            },
            "tags": {
              "type": "nested",
              "properties": {
                "tagType": {
                  "type": "keyword"
                },
                "value": {
                  "type": "keyword"
                },
                "key": {
                  "type": "keyword"
                }
              }
            }
          }
        },
        "references": {
          "type": "nested",
          "properties": {
            "spanID": {
              "type": "keyword"
            },
            "traceID": {
              "type": "keyword"
            },
            "refType": {
              "type": "keyword"
            }
          }
        },
        "startTimeMillis": {
          "type": "date"
        },
        "flags": {
          "type": "integer"
        },
        "operationName": {
          "type": "keyword"
        },
        "parentSpanID": {
          "type": "keyword"
        },
        "tags": {
          "type": "nested",
          "properties": {
            "tagType": {
              "type": "keyword"
            },
            "value": {
              "type": "keyword"
            },
            "key": {
              "type": "keyword"
            }
          }
        },
        "duration": {
          "type": "long"
        },
        "spanID": {
          "type": "keyword"
        },
        "startTime": {
          "type": "long"
        },
        "tag": {
          "type": "object",
          "properties": {}
        },
        "logs": {
          "type": "nested",
          "properties": {
            "fields": {
              "type": "nested",
              "properties": {
                "tagType": {
                  "type": "keyword"
                },
                "value": {
                  "type": "keyword"
                },
                "key": {
                  "type": "keyword"
                }
              }
            },
            "timestamp": {
              "type": "long"
            }
          }
        }
      }
    }
  },
  "composed_of": []
}

Services Composable Template (jaeger-service.tpl.json)

{
  "version": 3,
  "index_patterns": [
    "*jaeger-service-*"
  ],
  "priority": 102,
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "jaeger",
          "rollover_alias": "jaeger-service-write"
        },
        "mapping": {
          "nested_fields": {
            "limit": "50"
          }
        },
        "requests": {
          "cache": {
            "enable": "true"
          }
        },
        "number_of_shards": 5
      }
    },
    "aliases": {
      "jaeger-service-read": {}
    },
    "mappings": {
      "_routing": {
        "required": false
      },
      "numeric_detection": false,
      "dynamic_date_formats": [
        "strict_date_optional_time",
        "yyyy/MM/dd HH:mm:ss Z||yyyy/MM/dd Z"
      ],
      "dynamic": true,
      "_source": {
        "excludes": [],
        "includes": [],
        "enabled": true
      },
      "dynamic_templates": [
        {
          "span_tags_map": {
            "path_match": "tag.*",
            "mapping": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        },
        {
          "process_tags_map": {
            "path_match": "process.tag.*",
            "mapping": {
              "ignore_above": 256,
              "type": "keyword"
            }
          }
        }
      ],
      "date_detection": true,
      "properties": {
        "operationName": {
          "type": "keyword"
        },
        "serviceName": {
          "type": "keyword"
        }
      }
    }
  },
  "composed_of": []
}

Spans rollover origin index (jaeger-span-000001.idx.json)

Since it's not possible to add rollover writable aliases using templates, you'll have to deploy the origin index manually.

{
    "aliases": {
        "jaeger-span-write": {
            "is_write_index": true
        }
    }
}

Services rollover origin index (jaeger-service-000001.idx.json)

{
    "aliases": {
        "jaeger-service-write": {
            "is_write_index": true
        }
    }
}

K8s Jaeger Operator

If you're using the k8s operator to deploy your Jaeger stack, the all-in-one distribution won't work since it'll try to create the rollover job because of the necessary flag use-aliases: true. You'll have to use the version of the separate components. This is a basic setup using Jaeger CRD manifest:

apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger
spec:
  collector:
    options:
      es:
        use-aliases: true
        use-ilm: true
  query:
    options:
      es:
        use-aliases: true
        use-ilm: true
  storage:
    type: elasticsearch
    options:
      es:
        version: 7
        create-index-templates: false
        server-urls: http://elasticsearch:9200

  # Necessary for elasticsearch integration to work properly
  # read more: https://github.com/jaegertracing/jaeger-operator/issues/1187
  strategy: production

[jicki]

hope to support elasticsearch 8.2.x

[nonamef]

Elasticsearch is up to 8.5.0 and still waiting on an update? Using out of date infrastructure in production is becoming a burden.

[jkowall]

Opensearch is a good alternative. Not sure there is interest in building support for 8.x+. There are also other backends which are efficient as options.

[PetrusZ]

We are experiencing problems connecting Jaeger to our existing ES setup due to this issue. I've gone ahead and followed the work around instructions above, currently using the Helm chart for setup.

Ex:

   cmdlineParams:
      es.version: 7
      es.create-index-templates: false 

But no matter I still see this error relating to the template index patterns:

"caller":"./main.go:69","msg":"Failed to create span writer","error":"elastic: Error 400 (Bad Request): legacy template [jaeger-span] has index patterns [*jaeger-span-*] matching patterns from existing composable templates

I have also tried removing the leading * from the templates, but the error message stayed the same which leads me to believe this is still on the Jaeger end of things.

Anyone have further workaround info/updates on ES8 support?

try to use env as workaroud:

    extraEnv:
      - name: ES_VERSION
        value: "7"
      - name: ES_CREATE_INDEX_TEMPLATES
        value: "false"

[guitcastro]

mangli

This is a bug in the helm char: False property are treat as true.

https://github.com/jaegertracing/helm-charts/blob/main/charts/jaeger/templates/collector-deploy.yaml#L53

[rajims07]

which value we need to modify to work properly

[guitcastro]

which value we need to modify to work properly

As workaround? Just use "false" instead of false.

To fix the chart, check if values are empty or null instead of false:

The bugged code:

          {{- if $value }}
          - --{{ $key }}={{ $value }}
          {{- else }}
          - --{{ $key }}

[sheng-jie]

Any progress?