zipkin-aws
zipkin-aws copied to clipboard
openzipkin
zipkin aws storage elasticsearch not working with aws elsticsearch version 7+
I am using elasticsearch 7.10 image on aws. When I connect to elasticsearch I use the openzipkin/zipkin-aws:0.23 image. and i get an error:
DEBUG 1 --- [orker-epoll-2-2] z.s.i.ZipkinHttpCollector : Cannot decode spans due to IllegalArgumentException(Expected value at line 1 column 1 path $ reading List<Span> from json)
java.lang.IllegalArgumentException: Expected value at line 1 column 1 path $ reading List<Span> from json
at zipkin2.internal.JsonCodec.exceptionReading(JsonCodec.java:230) ~[zipkin-2.21.7.jar:?]
at zipkin2.internal.JsonCodec.readList(JsonCodec.java:151) ~[zipkin-2.21.7.jar:?]
at zipkin2.codec.SpanBytesDecoder$3.decodeList(SpanBytesDecoder.java:127) ~[zipkin-2.21.7.jar:?]
at zipkin2.codec.SpanBytesDecoder.doDecodeList(SpanBytesDecoder.java:209) ~[zipkin-2.21.7.jar:?]
at zipkin2.codec.SpanBytesDecoder$3.decodeList(SpanBytesDecoder.java:143) ~[zipkin-2.21.7.jar:?]
at zipkin2.collector.Collector.acceptSpans(Collector.java:149) ~[zipkin-collector-2.21.7.jar:?]
at zipkin2.server.internal.ZipkinHttpCollector.lambda$validateAndStoreSpans$0(ZipkinHttpCollector.java:154) ~[classes/:?]
at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) [?:?]
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) [?:?]
at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) [?:?]
at java.util.concurrent.CompletableFuture.complete(Unknown Source) [?:?]
at com.linecorp.armeria.common.HttpMessageAggregator.onComplete(HttpMessageAggregator.java:92) [armeria-0.99.9.jar:?]
at com.linecorp.armeria.common.stream.AbstractStreamMessage$CloseEvent.notifySubscriber(AbstractStreamMessage.java:269) [armeria-0.99.9.jar:?]
at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent0(DefaultStreamMessage.java:264) [armeria-0.99.9.jar:?]
at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent(DefaultStreamMessage.java:256) [armeria-0.99.9.jar:?]
at com.linecorp.armeria.common.stream.DefaultStreamMessage.handleCloseEvent(DefaultStreamMessage.java:436) [armeria-0.99.9.jar:?]
at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber0(DefaultStreamMessage.java:368) [armeria-0.99.9.jar:?]
at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber(DefaultStreamMessage.java:315) [armeria-0.99.9.jar:?]
at com.linecorp.armeria.common.stream.DefaultStreamMessage.addObjectOrEvent(DefaultStreamMessage.java:301) [armeria-0.99.9.jar:?]
at com.linecorp.armeria.common.stream.DefaultStreamMessage.close(DefaultStreamMessage.java:443) [armeria-0.99.9.jar:?]
at com.linecorp.armeria.server.Http1RequestDecoder.channelRead(Http1RequestDecoder.java:253) [armeria-0.99.9.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.51.Final.jar:4.1.51.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152) [netty-handler-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.51.Final.jar:4.1.51.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.51.Final.jar:4.1.51.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.51.Final.jar:4.1.51.Final]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: zipkin2.internal.gson.stream.MalformedJsonException: Expected value at line 1 column 1 path $
at zipkin2.internal.gson.stream.JsonReader.syntaxError(JsonReader.java:1564) ~[zipkin-2.21.7.jar:?]
at zipkin2.internal.gson.stream.JsonReader.doPeek(JsonReader.java:591) ~[zipkin-2.21.7.jar:?]
at zipkin2.internal.gson.stream.JsonReader.beginArray(JsonReader.java:344) ~[zipkin-2.21.7.jar:?]
at zipkin2.internal.JsonCodec$JsonReader.beginArray(JsonCodec.java:55) ~[zipkin-2.21.7.jar:?]
at zipkin2.internal.JsonCodec.readList(JsonCodec.java:145) ~[zipkin-2.21.7.jar:?]
Such problems were not noticed with the use of elasticsearch aws version 6.8
I suspect that this may be due to the fact that in elasticsearch 7+ Indices prior to 7.0 could contain a colon (:), but that’s been deprecated and won’t be supported in 7.0+ https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html
Ping @xeraa
José Carlos Chávez
fre. 21. jan. 2022 kl. 07:35 skrev ax-korneenkov @.***>:
I am using elasticsearch 7.10 image on aws. When I connect to elasticsearch I use the openzipkin/zipkin-aws:0.23 image. and i get an error:
DEBUG 1 --- [orker-epoll-2-2] z.s.i.ZipkinHttpCollector : Cannot decode spans due to IllegalArgumentException(Expected value at line 1 column 1 path $ reading List<Span> from json)
java.lang.IllegalArgumentException: Expected value at line 1 column 1 path $ reading List<Span> from json
at zipkin2.internal.JsonCodec.exceptionReading(JsonCodec.java:230) ~[zipkin-2.21.7.jar:?]
at zipkin2.internal.JsonCodec.readList(JsonCodec.java:151) ~[zipkin-2.21.7.jar:?]
at zipkin2.codec.SpanBytesDecoder$3.decodeList(SpanBytesDecoder.java:127) ~[zipkin-2.21.7.jar:?]
at zipkin2.codec.SpanBytesDecoder.doDecodeList(SpanBytesDecoder.java:209) ~[zipkin-2.21.7.jar:?]
at zipkin2.codec.SpanBytesDecoder$3.decodeList(SpanBytesDecoder.java:143) ~[zipkin-2.21.7.jar:?]
at zipkin2.collector.Collector.acceptSpans(Collector.java:149) ~[zipkin-collector-2.21.7.jar:?]
at zipkin2.server.internal.ZipkinHttpCollector.lambda$validateAndStoreSpans$0(ZipkinHttpCollector.java:154) ~[classes/:?]
at java.util.concurrent.CompletableFuture.uniHandle(Unknown Source) [?:?]
at java.util.concurrent.CompletableFuture$UniHandle.tryFire(Unknown Source) [?:?]
at java.util.concurrent.CompletableFuture.postComplete(Unknown Source) [?:?]
at java.util.concurrent.CompletableFuture.complete(Unknown Source) [?:?]
at com.linecorp.armeria.common.HttpMessageAggregator.onComplete(HttpMessageAggregator.java:92) [armeria-0.99.9.jar:?]
at com.linecorp.armeria.common.stream.AbstractStreamMessage$CloseEvent.notifySubscriber(AbstractStreamMessage.java:269) [armeria-0.99.9.jar:?]
at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent0(DefaultStreamMessage.java:264) [armeria-0.99.9.jar:?]
at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent(DefaultStreamMessage.java:256) [armeria-0.99.9.jar:?]
at com.linecorp.armeria.common.stream.DefaultStreamMessage.handleCloseEvent(DefaultStreamMessage.java:436) [armeria-0.99.9.jar:?]
at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber0(DefaultStreamMessage.java:368) [armeria-0.99.9.jar:?]
at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber(DefaultStreamMessage.java:315) [armeria-0.99.9.jar:?]
at com.linecorp.armeria.common.stream.DefaultStreamMessage.addObjectOrEvent(DefaultStreamMessage.java:301) [armeria-0.99.9.jar:?]
at com.linecorp.armeria.common.stream.DefaultStreamMessage.close(DefaultStreamMessage.java:443) [armeria-0.99.9.jar:?]
at com.linecorp.armeria.server.Http1RequestDecoder.channelRead(Http1RequestDecoder.java:253) [armeria-0.99.9.jar:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:324) [netty-codec-4.1.51.Final.jar:4.1.51.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:296) [netty-codec-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:271) [netty-handler-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152) [netty-handler-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.51.Final.jar:4.1.51.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:792) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:475) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [netty-transport-native-epoll-4.1.51.Final-linux-x86_64.jar:4.1.51.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.51.Final.jar:4.1.51.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.51.Final.jar:4.1.51.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.51.Final.jar:4.1.51.Final]
at java.lang.Thread.run(Unknown Source) [?:?]
Caused by: zipkin2.internal.gson.stream.MalformedJsonException: Expected value at line 1 column 1 path $
at zipkin2.internal.gson.stream.JsonReader.syntaxError(JsonReader.java:1564) ~[zipkin-2.21.7.jar:?]
at zipkin2.internal.gson.stream.JsonReader.doPeek(JsonReader.java:591) ~[zipkin-2.21.7.jar:?]
at zipkin2.internal.gson.stream.JsonReader.beginArray(JsonReader.java:344) ~[zipkin-2.21.7.jar:?]
at zipkin2.internal.JsonCodec$JsonReader.beginArray(JsonCodec.java:55) ~[zipkin-2.21.7.jar:?]
at zipkin2.internal.JsonCodec.readList(JsonCodec.java:145) ~[zipkin-2.21.7.jar:?]
Such problems were not noticed with the use of elasticsearch aws version 6.8
I suspect that this may be due to the fact that in elasticsearch 7+ Indices prior to 7.0 could contain a colon (:), but that’s been deprecated and won’t be supported in 7.0+
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-create-index.html
— Reply to this email directly, view it on GitHub https://github.com/openzipkin/zipkin-aws/issues/196, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXOYATIESIM5JUO6ECL6VTUXD5E3ANCNFSM5MOWFOSA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
You are receiving this because you are subscribed to this thread.Message ID: @.***>
- Yes, you cannot create indices with a colon in 7.0+ any more but you can still read from it.
- There was an earlier discussion on it in https://github.com/openzipkin/zipkin/issues/2219 and I thought this was generally fixed? Also 7.0 came out like 2 years ago, so there must be something special / uncommon going on here.
Hello, we can't upgrade our ELK-Cluter from version 6 to 7 due to this ":"-restriction. Is there any chance to receive an update for this issue. Idealy with a fix, so zipkin cat create an Index in Elastic >= 7.0?
@xf0e As @xeraa mentioned, it's not a limitation in Zipkin; it is a restriction in Elasticsearch that an index cannot be created with : from Elasticsearch 7. You will have to use a different delimiter. As far as I remember from the outcome of the issue @xeraa linked, Zipkin should use a different delimiter when writing to Elasticsearch >= 7.0.
@ax-korneenkov that stacktrace is showing that an exception is being thrown by the collector, not the storage layer. It is failing to decode spans sent to the Zipkin collector. Make sure that spans are being sent in the correct format to the correct endpoint.
I think @shakuzen was correct. maybe this is the wrong input data. in any case, if this were an issue it should be on the zipkin repo, which has been significantly updated recently
