[Bug]: Jaeger Query UI doesn't show traces

[simul-tion]

What happened?

I have setup Jaeger query and collector(latest version) on Kubernetes with ES as storage(version : 6.8.22) backend. The problem that I'm facing right now is Jaeger Query doesn't show any other service except Jaeger Query in the dropdown. So no trace information get reflected in Jaeger Query while I see events stored in Elastic Search. There is nothing in the logs of Jaeger query or collector which points to any problems, logs for these components as shown below. Please suggest any pointers on where can I begin to troubleshoot this.

Steps to reproduce

1. Setup Otel collector, Jaeger components (Collector and query not as part of allInOne but as independent pods), using latest image.
2. Setup ES version : 6.8.22
3. Use Local exporter example from link - https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/troubleshooting.md

Expected behavior

I can see logs printing trace information, ES has trace information but Jaeger query doesn't show traces in UI.

Relevant log output

Nothing in Jaeger Query or Collector logs:

Jaeger Query Logs:

{"level":"info","ts":1662628828.5362358,"caller":"channelz/funcs.go:340","msg":"[core][Channel #2] parsed dial target is: {Scheme:passthrough Authority: Endpoint::16685 URL:{Scheme:passthrough Opaque: User: Host: Path:/:16685 RawPath: ForceQuery:false RawQuery: Fragment: RawFragment:}}","system":"grpc","grpc_log":true}
{"level":"info","ts":1662628828.536245,"caller":"channelz/funcs.go:340","msg":"[core][Channel #2] Channel authority set to "localhost:16685"","system":"grpc","grpc_log":true}
{"level":"info","ts":1662628828.5363364,"caller":"channelz/funcs.go:340","msg":"[core][Channel #2] Resolver state updated: {\n "Addresses": [\n {\n "Addr": ":16685",\n "ServerName": "",\n "Attributes": null,\n "BalancerAttributes": null,\n "Type": 0,\n "Metadata": null\n }\n ],\n "ServiceConfig": null,\n "Attributes": null\n} (resolver returned new addresses)","system":"grpc","grpc_log":true}
{"level":"info","ts":1662628828.5363655,"caller":"channelz/funcs.go:340","msg":"[core][Channel #2] Channel switches to new LB policy "pick_first"","system":"grpc","grpc_log":true}
{"level":"info","ts":1662628828.5363824,"caller":"channelz/funcs.go:340","msg":"[core][Channel #2 SubChannel #3] Subchannel created","system":"grpc","grpc_log":true}
{"level":"info","ts":1662628828.536771,"caller":"app/static_handler.go:181","msg":"UI config path not provided, config file will not be watched"}
{"level":"info","ts":1662628828.5368516,"caller":"app/server.go:217","msg":"Query server started","http_addr":"[::]:16686","grpc_addr":"[::]:16685"}
{"level":"info","ts":1662628828.5368638,"caller":"healthcheck/handler.go:129","msg":"Health Check state change","status":"ready"}
{"level":"info","ts":1662628828.536879,"caller":"app/server.go:300","msg":"Starting GRPC server","port":16685,"addr":":16685"}
{"level":"info","ts":1662628828.5368896,"caller":"channelz/funcs.go:340","msg":"[core][Server #1 ListenSocket #4] ListenSocket created","system":"grpc","grpc_log":true}
{"level":"info","ts":1662628828.5369315,"caller":"channelz/funcs.go:340","msg":"[core][Channel #2 SubChannel #3] Subchannel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
{"level":"info","ts":1662628828.5369415,"caller":"channelz/funcs.go:340","msg":"[core][Channel #2 SubChannel #3] Subchannel picks a new address ":16685" to connect","system":"grpc","grpc_log":true}
{"level":"info","ts":1662628828.5370312,"caller":"app/server.go:281","msg":"Starting HTTP server","port":16686,"addr":":16686"}
{"level":"info","ts":1662628828.5370584,"caller":"grpclog/component.go:71","msg":"[core]pickfirstBalancer: UpdateSubConnState: 0xc000505be0, {CONNECTING }","system":"grpc","grpc_log":true}
{"level":"info","ts":1662628828.537072,"caller":"channelz/funcs.go:340","msg":"[core][Channel #2] Channel Connectivity change to CONNECTING","system":"grpc","grpc_log":true}
{"level":"info","ts":1662628828.5372539,"caller":"channelz/funcs.go:340","msg":"[core][Channel #2 SubChannel #3] Subchannel Connectivity change to READY","system":"grpc","grpc_log":true}
{"level":"info","ts":1662628828.5372684,"caller":"grpclog/component.go:71","msg":"[core]pickfirstBalancer: UpdateSubConnState: 0xc000505be0, {READY }","system":"grpc","grpc_log":true}
{"level":"info","ts":1662628828.5372767,"caller":"channelz/funcs.go:340","msg":"[core][Channel #2] Channel Connectivity change to READY","system":"grpc","grpc_log":true}

Jaeger Collector Logs

2022/09/08 09:20:28 maxprocs: Leaving GOMAXPROCS=2: CPU quota undefined
{"level":"info","ts":1662628828.426277,"caller":"flags/service.go:119","msg":"Mounting metrics handler on admin server","route":"/metrics"}
{"level":"info","ts":1662628828.4263563,"caller":"flags/service.go:125","msg":"Mounting expvar handler on admin server","route":"/debug/vars"}
{"level":"info","ts":1662628828.4273946,"caller":"flags/admin.go:128","msg":"Mounting health check on admin server","route":"/"}
{"level":"info","ts":1662628828.4274535,"caller":"flags/admin.go:141","msg":"Starting admin HTTP server","http-addr":":14269"}
{"level":"info","ts":1662628828.4274795,"caller":"flags/admin.go:120","msg":"Admin server started","http.host-port":"[::]:14269","health-status":"unavailable"}
{"level":"info","ts":1662628828.4574142,"caller":"config/config.go:206","msg":"Elasticsearch detected","version":6}
{"level":"info","ts":1662628828.4876025,"caller":"static/strategy_store.go:203","msg":"No sampling strategies provided or URL is unavailable, using defaults"}
{"level":"info","ts":1662628828.5252645,"caller":"channelz/funcs.go:340","msg":"[core][Server #1] Server created","system":"grpc","grpc_log":true}
{"level":"info","ts":1662628828.5253756,"caller":"server/grpc.go:104","msg":"Starting jaeger-collector gRPC server","grpc.host-port":"[::]:14250"}
{"level":"info","ts":1662628828.5253954,"caller":"server/http.go:48","msg":"Starting jaeger-collector HTTP server","http host-port":":14268"}
{"level":"info","ts":1662628828.5254807,"caller":"channelz/funcs.go:340","msg":"[core][Server #1 ListenSocket #2] ListenSocket created","system":"grpc","grpc_log":true}
{"level":"info","ts":1662628828.525679,"caller":"server/zipkin.go:55","msg":"Listening for Zipkin HTTP traffic","zipkin host-port":":9411"}
{"level":"info","ts":1662628828.5577762,"caller":"channelz/funcs.go:340","msg":"[core][Server #3] Server created","system":"grpc","grpc_log":true}
{"level":"info","ts":1662628828.557808,"caller":"otlpreceiver/otlp.go:70","msg":"Starting GRPC server on endpoint 0.0.0.0:4317"}
{"level":"info","ts":1662628828.557865,"caller":"otlpreceiver/otlp.go:88","msg":"Starting HTTP server on endpoint 0.0.0.0:4318"}
{"level":"info","ts":1662628828.5578928,"caller":"healthcheck/handler.go:129","msg":"Health Check state change","status":"ready"}
{"level":"info","ts":1662628828.5579045,"caller":"channelz/funcs.go:340","msg":"[core][Server #3 ListenSocket #4] ListenSocket created","system":"grpc","grpc_log":true}


Otel Collector Logs:

2022/09/08 12:20:34 proto: duplicate proto type registered: jaeger.api_v2.PostSpansRequest
2022/09/08 12:20:34 proto: duplicate proto type registered: jaeger.api_v2.PostSpansResponse
2022-09-08T12:20:35.000Z	info	service/telemetry.go:115	Setting up own telemetry...
2022-09-08T12:20:35.000Z	info	service/telemetry.go:156	Serving Prometheus metrics	{"address": "0.0.0.0:8888", "level": "basic"}
2022-09-08T12:20:35.000Z	info	components/components.go:30	In development component. May change in the future.	{"kind": "exporter", "data_type": "logs", "name": "logging", "stability": "in development"}
2022-09-08T12:20:35.000Z	info	memorylimiterprocessor/memorylimiter.go:113	Memory limiter configured	{"kind": "processor", "name": "memory_limiter", "pipeline": "logs", "limit_mib": 819, "spike_limit_mib": 256, "check_interval": 5}
2022-09-08T12:20:35.000Z	info	components/components.go:30	In development component. May change in the future.	{"kind": "exporter", "data_type": "traces", "name": "logging", "stability": "in development"}
2022-09-08T12:20:35.001Z	info	components/components.go:30	In development component. May change in the future.	{"kind": "exporter", "data_type": "metrics", "name": "logging", "stability": "in development"}
2022-09-08T12:20:35.001Z	info	service/service.go:112	Starting otelcol-contrib...	{"Version": "0.59.0", "NumCPU": 2}
2022-09-08T12:20:35.001Z	info	extensions/extensions.go:42	Starting extensions...
2022-09-08T12:20:35.001Z	info	extensions/extensions.go:45	Extension is starting...	{"kind": "extension", "name": "health_check"}
2022-09-08T12:20:35.001Z	info	[email protected]/healthcheckextension.go:44	Starting health_check extension	{"kind": "extension", "name": "health_check", "config": {"Endpoint":"0.0.0.0:13133","TLSSetting":null,"CORS":null,"Auth":null,"MaxRequestBodySize":0,"IncludeMetadata":false,"Path":"/","CheckCollectorPipeline":{"Enabled":false,"Interval":"5m","ExporterFailureThreshold":5}}}
2022-09-08T12:20:35.001Z	info	extensions/extensions.go:49	Extension started.	{"kind": "extension", "name": "health_check"}
2022-09-08T12:20:35.001Z	info	extensions/extensions.go:45	Extension is starting...	{"kind": "extension", "name": "memory_ballast"}
2022-09-08T12:20:35.002Z	info	ballastextension/memory_ballast.go:52	Setting memory ballast	{"kind": "extension", "name": "memory_ballast", "MiBs": 0}
2022-09-08T12:20:35.002Z	info	extensions/extensions.go:49	Extension started.	{"kind": "extension", "name": "memory_ballast"}
2022-09-08T12:20:35.002Z	info	pipelines/pipelines.go:74	Starting exporters...
2022-09-08T12:20:35.002Z	info	pipelines/pipelines.go:78	Exporter is starting...	{"kind": "exporter", "data_type": "metrics", "name": "logging"}
2022-09-08T12:20:35.002Z	info	pipelines/pipelines.go:82	Exporter started.	{"kind": "exporter", "data_type": "metrics", "name": "logging"}
2022-09-08T12:20:35.002Z	info	pipelines/pipelines.go:78	Exporter is starting...	{"kind": "exporter", "data_type": "logs", "name": "logging"}
2022-09-08T12:20:35.003Z	info	pipelines/pipelines.go:82	Exporter started.	{"kind": "exporter", "data_type": "logs", "name": "logging"}
2022-09-08T12:20:35.003Z	info	pipelines/pipelines.go:78	Exporter is starting...	{"kind": "exporter", "data_type": "traces", "name": "logging"}
2022-09-08T12:20:35.003Z	info	pipelines/pipelines.go:82	Exporter started.	{"kind": "exporter", "data_type": "traces", "name": "logging"}
2022-09-08T12:20:35.003Z	info	pipelines/pipelines.go:78	Exporter is starting...	{"kind": "exporter", "data_type": "traces", "name": "jaeger"}
2022-09-08T12:20:35.003Z	info	[email protected]/exporter.go:186	State of the connection with the Jaeger Collector backend	{"kind": "exporter", "data_type": "traces", "name": "jaeger", "state": "IDLE"}
2022-09-08T12:20:35.003Z	info	pipelines/pipelines.go:82	Exporter started.	{"kind": "exporter", "data_type": "traces", "name": "jaeger"}
2022-09-08T12:20:35.003Z	info	pipelines/pipelines.go:86	Starting processors...

Screenshot

No response

Additional context

No response

Jaeger backend version

Latest - 1.37

SDK

No response

Pipeline

Local Exporter -> Otel Collector -> Jaeger Collector -> ES -> Jaeger Query

Stogage backend

Elastic Search 6.8.22

Operating system

No response

Deployment model

Kubernetes

Deployment configs

No response