Zipkin with ES7 - Cannot invoke "String.hashCode()" because "<local3>" is null

[yzdann]

Zipkin2 with ES7 - Cannot invoke "String.hashCode()" because "" is null

Hi! We have this issue since we upgrade our zipkin backend storage from es6 to es7.

Cannot invoke "String.hashCode()" because "" is null

2021-12-06 14:46:10.488 WARN [/] 1 --- [orker-epoll-2-1] z.s.i.BodyIsExceptionMessage : Unexpected error handling request.
java.lang.NullPointerException: Cannot invoke "String.hashCode()" because "<local3>" is null
at zipkin2.elasticsearch.internal.JsonSerializers.parseSpan(JsonSerializers.java:67) ~[zipkin-storage-elasticsearch-2.23.9.jar:?]
at zipkin2.elasticsearch.internal.client.SearchResultConverter.convert(SearchResultConverter.java:46) ~[zipkin-storage-elasticsearch-2.23.9.jar:?]
at zipkin2.elasticsearch.internal.client.SearchResultConverter.convert(SearchResultConverter.java:27) ~[zipkin-storage-elasticsearch-2.23.9.jar:?]
at zipkin2.elasticsearch.internal.client.HttpCall.parseResponse(HttpCall.java:275) ~[zipkin-storage-elasticsearch-2.23.9.jar:?]
at zipkin2.elasticsearch.internal.client.HttpCall.doExecute(HttpCall.java:166) ~[zipkin-storage-elasticsearch-2.23.9.jar:?]
at zipkin2.Call$Base.execute(Call.java:391) ~[zipkin-2.23.9.jar:?]
at zipkin2.Call$FlatMapping.doExecute(Call.java:276) ~[zipkin-2.23.9.jar:?]
at zipkin2.Call$Base.execute(Call.java:391) ~[zipkin-2.23.9.jar:?]
at zipkin2.Call$Mapping.doExecute(Call.java:237) ~[zipkin-2.23.9.jar:?]
at zipkin2.Call$Base.execute(Call.java:391) ~[zipkin-2.23.9.jar:?]
at zipkin2.Call$Mapping.doExecute(Call.java:237) ~[zipkin-2.23.9.jar:?]
at zipkin2.Call$Base.execute(Call.java:391) ~[zipkin-2.23.9.jar:?]
at zipkin2.server.internal.ZipkinQueryApiV2.getTraces(ZipkinQueryApiV2.java:147) ~[classes/:?]
at com.linecorp.armeria.internal.server.annotation.AnnotatedService.invoke(AnnotatedService.java:391) ~[armeria-1.13.3.jar:?]
at com.linecorp.armeria.internal.server.annotation.AnnotatedService.lambda$serve0$8(AnnotatedService.java:359) ~[armeria-1.13.3.jar:?]
at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?]
at java.util.concurrent.CompletableFuture$Completion.run(Unknown Source) ~[?:?]
at com.linecorp.armeria.common.RequestContext.lambda$makeContextAware$3(RequestContext.java:547) ~[armeria-1.13.3.jar:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?]
at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.69.Final.jar:4.1.69.Final]
at java.lang.Thread.run(Unknown Source) [?:?]

Steps to Reproduce

Zipkin: ghcr.io/openzipkin/zipkin:2 ES: 7.13.4

[jcchavezs]

Ping @xeraa

On Mon, Dec 6, 2021, 16:07 Yazdan @.***> wrote:

Zipkin2 with ES7 - Cannot invoke "String.hashCode()" because "" is null

Hi! We have this issue since we upgrade our zipkin backend storage from es6 to es7.

Cannot invoke "String.hashCode()" because "" is null [image: ss2] https://user-images.githubusercontent.com/22779039/144869527-5abcef82-a664-4392-9595-5a30853991f9.png

[image: ss] https://user-images.githubusercontent.com/22779039/144869524-6e7a9bef-fca6-44d3-becf-13b30902763a.png

2021-12-06 14:46:10.488 WARN [/] 1 --- [orker-epoll-2-1] z.s.i.BodyIsExceptionMessage : Unexpected error handling request.java.lang.NullPointerException: Cannot invoke "String.hashCode()" because "" is null at zipkin2.elasticsearch.internal.JsonSerializers.parseSpan(JsonSerializers.java:67) ~[zipkin-storage-elasticsearch-2.23.9.jar:?] at zipkin2.elasticsearch.internal.client.SearchResultConverter.convert(SearchResultConverter.java:46) ~[zipkin-storage-elasticsearch-2.23.9.jar:?] at zipkin2.elasticsearch.internal.client.SearchResultConverter.convert(SearchResultConverter.java:27) ~[zipkin-storage-elasticsearch-2.23.9.jar:?] at zipkin2.elasticsearch.internal.client.HttpCall.parseResponse(HttpCall.java:275) ~[zipkin-storage-elasticsearch-2.23.9.jar:?] at zipkin2.elasticsearch.internal.client.HttpCall.doExecute(HttpCall.java:166) ~[zipkin-storage-elasticsearch-2.23.9.jar:?] at zipkin2.Call$Base.execute(Call.java:391) ~[zipkin-2.23.9.jar:?] at zipkin2.Call$FlatMapping.doExecute(Call.java:276) ~[zipkin-2.23.9.jar:?] at zipkin2.Call$Base.execute(Call.java:391) ~[zipkin-2.23.9.jar:?] at zipkin2.Call$Mapping.doExecute(Call.java:237) ~[zipkin-2.23.9.jar:?] at zipkin2.Call$Base.execute(Call.java:391) ~[zipkin-2.23.9.jar:?] at zipkin2.Call$Mapping.doExecute(Call.java:237) ~[zipkin-2.23.9.jar:?] at zipkin2.Call$Base.execute(Call.java:391) ~[zipkin-2.23.9.jar:?] at zipkin2.server.internal.ZipkinQueryApiV2.getTraces(ZipkinQueryApiV2.java:147) ~[classes/:?] at com.linecorp.armeria.internal.server.annotation.AnnotatedService.invoke(AnnotatedService.java:391) ~[armeria-1.13.3.jar:?] at com.linecorp.armeria.internal.server.annotation.AnnotatedService.lambda$serve0$8(AnnotatedService.java:359) ~[armeria-1.13.3.jar:?] at java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source) ~[?:?] at java.util.concurrent.CompletableFuture$Completion.run(Unknown Source) ~[?:?] at com.linecorp.armeria.common.RequestContext.lambda$makeContextAware$3(RequestContext.java:547) ~[armeria-1.13.3.jar:?] at java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source) ~[?:?] at java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?] at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.69.Final.jar:4.1.69.Final] at java.lang.Thread.run(Unknown Source) [?:?]

Steps to Reproduce

Zipkin: ghcr.io/openzipkin/zipkin:2 ES: 7.13.4

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/openzipkin/zipkin/issues/3400, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXOYAV6DIOJIISOPRRVANDUPTGSBANCNFSM5JO3GTWQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[xeraa]

Hmmm. If I'm not mistaken, Zipkin has written it's own Elasticsearch client, so my knowledge here will be very shallow.

I assume we're talking about https://github.com/openzipkin/zipkin/blob/master/zipkin-storage/elasticsearch/src/main/java/zipkin2/elasticsearch/internal/JsonSerializers.java#L67? I don't think that's really on the Elasticsearch side (which is not a solution, I know 😕)

[yzdann]

More details:

Response:

Indices:

Role (We gave more than what it needs!):

Mappings:

[xeraa]

Yes, I don't think this is an Elasticsearch issue — you can write and read data. The problem is somewhere in the Zipkin codebase visualizing the data and it runs into a NPE (maybe some missing field; not exactly sure).

[yzdann]

I removed ILM from Zipkin indices, and everything works fine!

[xeraa]

Interesting. Did you use a specific policy? Maybe making the index read-only was causing issues? But it's just a wild guess.

[MonsterOli]

FYI, there's a fix waiting in PR for this issue: https://github.com/openzipkin/zipkin/pull/3437

[yzdann]

Interesting. Did you use a specific policy? Maybe making the index read-only was causing issues? But it's just a wild guess.

Sorry for my late response. I don't have access to that environment anymore.

[codefromthecrypt]

closing this as it is very out-of-date, and the related issue hinted at a non-standard setup.