Jaeger collector fails to parse elastic search URL

[teoyaomiqui]

Describe the bug I am using jaeger-collector installed via jaeger helm char, the elasticsearch is preinstalled using AWS. I've used jaeger chart with a single component enabled - collector. I've tried passing elasticsearch env variables ES_SERVER_URLS but it appears to be ignored, I resorted to using command line flags es.server-urls. server-urls appear to be picked up by the collector service, however, it fails to parse the URL provided, and I am getting this error.

{"level":"fatal","ts":1627928859.771347,"caller":"command-line-arguments/main.go:75","msg":"Failed to init storage factory","error":"failed to create primary Elasticsearch client: parse \"\\\"https://search-<redacted>.ap-southeast-1.es.amazonaws.com:443\\\"\": first path segment in URL cannot contain colon","stacktrace":"main.main.func1\n\tcommand-line-arguments/main.go:75\ngithub.com/spf13/cobra.(*Command).execute\n\tgithub.com/spf13/[email protected]/command.go:838\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\tgithub.com/spf13/[email protected]/command.go:943\ngithub.com/spf13/cobra.(*Command).Execute\n\tgithub.com/spf13/[email protected]/command.go:883\nmain.main\n\tcommand-line-arguments/main.go:137\nruntime.main\n\truntime/proc.go:204"}

I've tried using go net/url.Parse(url) and it appears to be working fine with the URL

Here is the deployment itself

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "6"
    meta.helm.sh/release-name: jaeger
    meta.helm.sh/release-namespace: istio-system
  generation: 6
  labels:
    app.kubernetes.io/component: collector
    app.kubernetes.io/instance: jaeger
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: jaeger
    app.kubernetes.io/version: 1.22.0
    helm.sh/chart: jaeger-0.46.4
  name: jaeger-collector
  namespace: istio-system
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/component: collector
      app.kubernetes.io/instance: jaeger
      app.kubernetes.io/name: jaeger
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app.kubernetes.io/component: collector
        app.kubernetes.io/instance: jaeger
        app.kubernetes.io/name: jaeger
    spec:
      containers:
      - args:
        - --es.server-urls=https://search-<redacted>.ap-southeast-1.es.amazonaws.com:443
        - --es.index-prefix=jaeger-tracing-dev
        env:
        - name: COLLECTOR_ZIPKIN_HOST_PORT
          value: "9411"
        - name: SPAN_STORAGE_TYPE
          value: elasticsearch
        - name: ES_SERVER_URLS
          value: https://search-<redacted>.ap-southeast-1.es.amazonaws.com:443
        - name: ES_USERNAME
          value: elastic
        - name: ES_INDEX_PREFIX
          value: jaeger-tracing-dev
        image: jaegertracing/jaeger-collector:1.22.0
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /
            port: admin
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: jaeger-collector
        ports:
        - containerPort: 14250
          name: grpc
          protocol: TCP
        - containerPort: 14268
          name: http
          protocol: TCP
        - containerPort: 14269
          name: admin
          protocol: TCP
        - containerPort: 9411
          name: zipkin
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /
            port: admin
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources: {}
        securityContext: {}
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      serviceAccount: jaeger-collector
      serviceAccountName: jaeger-collector
      terminationGracePeriodSeconds: 30

It appears that Jaeger doesn't expect an actual URL but wants the hosts instead of URLs, I will double check whats going to happen if I don't specify schema

I've also tried to manually build a collector from cmd/collector , but It didn't even recognize the flag.

To Reproduce Steps to reproduce the behavior:

1. add flags to args field of the jaeger deployment

    spec:
      containers:
      - args:
        - --es.server-urls=https://search-***.ap-southeast-1.es.amazonaws.com:443

2. get the jaeger pod with kubectl get pods -A | grep jaeger
3. read jaeger logs kubectl logs

Expected behavior Jaeger connecting to Elasticsearch storage successfully

Version (please complete the following information):

• OS: Linux
• Jaeger version: 1.22
• Deployment: Kubernetes

[jpkrohling]

Is the Helm Chart doing something fancy here? The flag as presented does work for me. Note that the error below is because the host itself isn't reachable to me, which is past the point of the reported error:

$ pwd
/home/jpkroehling/Projects/src/github.com/jaegertracing/jaeger

$ SPAN_STORAGE_TYPE=elasticsearch go run ./cmd/collector/ --es.server-urls=https://search-abc.ap-southeast-1.es.amazonaws.com:443
2021/08/03 10:48:11 maxprocs: Leaving GOMAXPROCS=12: CPU quota undefined
{"level":"info","ts":1627980491.7870164,"caller":"flags/service.go:117","msg":"Mounting metrics handler on admin server","route":"/metrics"}
{"level":"info","ts":1627980491.7870436,"caller":"flags/service.go:123","msg":"Mounting expvar handler on admin server","route":"/debug/vars"}
{"level":"info","ts":1627980491.7872345,"caller":"flags/admin.go:105","msg":"Mounting health check on admin server","route":"/"}
{"level":"info","ts":1627980491.7872605,"caller":"flags/admin.go:111","msg":"Starting admin HTTP server","http-addr":":14269"}
{"level":"info","ts":1627980491.7872725,"caller":"flags/admin.go:97","msg":"Admin server started","http.host-port":"[::]:14269","health-status":"unavailable"}
^C{"level":"fatal","ts":1627980497.797236,"caller":"collector/main.go:76","msg":"Failed to init storage factory","error":"failed to create primary Elasticsearch client: Head \"https://search-abc.ap-southeast-1.es.amazonaws.com:443\": context deadline exceeded","stacktrace":"main.main.func1\n\t/home/jpkroehling/Projects/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:76\ngithub.com/spf13/cobra.(*Command).execute\n\t/home/jpkroehling/Projects/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:838\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/home/jpkroehling/Projects/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:943\ngithub.com/spf13/cobra.(*Command).Execute\n\t/home/jpkroehling/Projects/src/github.com/jaegertracing/jaeger/vendor/github.com/spf13/cobra/command.go:883\nmain.main\n\t/home/jpkroehling/Projects/src/github.com/jaegertracing/jaeger/cmd/collector/main.go:138\nruntime.main\n\t/home/jpkroehling/bin/go/src/runtime/proc.go:225"}

[mehta-ankit]

@teoyaomiqui Can you share your values.yaml file for the chart please.

I have a hunch that you are setting cmdlineParams for Collector but instead if you set storage.elasticsearch.host and port the helpers.tpl template will create the right ES_SERVER_URLS environment variable for the collector. Something like this

       - name: ES_SERVER_URLS
          value: http://elasticsearch-test-data:9200

Look here: https://github.com/jaegertracing/helm-charts/blob/main/charts/jaeger/templates/_helpers.tpl#L328 elasticsearch.env is used in storage.env here: https://github.com/jaegertracing/helm-charts/blob/main/charts/jaeger/templates/_helpers.tpl#L356 which gets used in collector deployment here: https://github.com/jaegertracing/helm-charts/blob/main/charts/jaeger/templates/collector-deploy.yaml#L86

Having your values file like this should work for you:

storage:
    type: elasticsearch
    elasticsearch:
       scheme: https
       host: search-<redacted>.ap-southeast-1.es.amazonaws.com
       port: 443