Get traces returning 500 response with response body set to `ResponseTimeoutException`

[msmsimondean]

Describe the Bug

I'm making a request to a URL that looks like this:

https://example.com/zipkin/api/v2/traces?serviceName={serviceName}&spanName={spanName}&endTs={endTs}&lookback={lookback}&limit={limit}

I've getting back a 500 status code with the following response body:

ResponseTimeoutException

It looks like Armeria might be timing out when calling Elasticsearch and for some reason the response body is being set to the simple name of the Armeria exception class. That's just a guess though. It's a really weird response body to be receiving 😄. Is there a way to configure the timeouts that Zipkin server uses when calling Elasticsearch?

Steps to Reproduce

See above

Expected Behaviour

To receive a valid JSON response body that contains an array of traces.

[jcchavezs]

Ping @anuraaga

On Thu, 6 May 2021, 00:41 Simon Dean, @.***> wrote:

Describe the Bug

I'm making a request to a URL that looks like this:

https://example.com/zipkin/api/v2/traces?serviceName={serviceName}&spanName={spanName}&endTs={endTs}&lookback={lookback}&limit={limit}

I've getting back a 500 status code with the following response body:

ResponseTimeoutException

It looks like Armeria might be timing out when calling Elasticsearch and for some reason the response body is being set to the simple name of the Armeria exception class. That's just a guess though. It's a really weird response body to be receiving 😄. Is there a way to configure the timeouts that Zipkin server uses when calling Elasticsearch? Steps to Reproduce

See above Expected Behaviour

To receive a valid JSON response body that contains an array of traces.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/openzipkin/zipkin/issues/3352, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXOYATKCGLJKDLSZ24W4D3TMHCQPANCNFSM44F4GESA .

[codeneno]

we met this bug too,please resolve this bug. `2021-06-01 08:17:50.404 WARN [/] 57889 --- [ing-tasks-1-853] z.s.i.BodyIsExceptionMessage : Unexpected error handling request.

com.linecorp.armeria.client.ResponseTimeoutException: null at com.linecorp.armeria.client.ResponseTimeoutException.get(ResponseTimeoutException.java:36) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.client.HttpResponseDecoder$HttpResponseWrapper.initTimeout(HttpResponseDecoder.java:381) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.client.HttpRequestSubscriber.operationComplete(HttpRequestSubscriber.java:120) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.client.HttpRequestSubscriber.operationComplete(HttpRequestSubscriber.java:54) ~[armeria-1.3.0.jar!/:?] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[netty-common-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[netty-common-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[netty-common-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184) ~[netty-common-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at com.linecorp.armeria.client.HttpRequestSubscriber.write(HttpRequestSubscriber.java:286) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.client.HttpRequestSubscriber.onComplete(HttpRequestSubscriber.java:260) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.common.stream.AbstractStreamMessage$CloseEvent.notifySubscriber(AbstractStreamMessage.java:270) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent0(DefaultStreamMessage.java:264) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriberOfCloseEvent(DefaultStreamMessage.java:256) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.common.stream.DefaultStreamMessage.handleCloseEvent(DefaultStreamMessage.java:436) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.common.stream.DefaultStreamMessage.notifySubscriber0(DefaultStreamMessage.java:368) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.common.stream.DefaultStreamMessage.doRequest(DefaultStreamMessage.java:239) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.common.stream.DefaultStreamMessage.request(DefaultStreamMessage.java:224) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.common.stream.AbstractStreamMessage$SubscriptionImpl.request(AbstractStreamMessage.java:224) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.client.HttpRequestSubscriber.operationComplete(HttpRequestSubscriber.java:128) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.client.HttpRequestSubscriber.operationComplete(HttpRequestSubscriber.java:54) ~[armeria-1.3.0.jar!/:?] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[netty-common-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[netty-common-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[netty-common-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:184) ~[netty-common-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:95) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:30) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at com.linecorp.armeria.client.HttpRequestSubscriber.writeFirstHeader(HttpRequestSubscriber.java:205) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.client.HttpRequestSubscriber.onSubscribe(HttpRequestSubscriber.java:181) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.common.stream.DefaultStreamMessage.subscribe(DefaultStreamMessage.java:139) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.common.stream.DefaultStreamMessage.subscribe(DefaultStreamMessage.java:126) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.common.stream.AbstractStreamMessage.subscribe(AbstractStreamMessage.java:74) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.common.stream.AbstractStreamMessage.subscribe(AbstractStreamMessage.java:65) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.common.HeaderOverridingHttpRequest.subscribe(HeaderOverridingHttpRequest.java:89) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.client.HttpSessionHandler.invoke(HttpSessionHandler.java:206) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.client.HttpClientDelegate.doExecute(HttpClientDelegate.java:276) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.client.HttpClientDelegate.lambda$acquireConnectionAndExecute$2(HttpClientDelegate.java:167) ~[armeria-1.3.0.jar!/:?] at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836) ~[?:1.8.0_291] at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811) ~[?:1.8.0_291] at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488) ~[?:1.8.0_291] at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975) ~[?:1.8.0_291] at com.linecorp.armeria.client.HttpChannelPool$ChannelAcquisitionFuture.complete(HttpChannelPool.java:823) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.client.HttpChannelPool.notifyConnect(HttpChannelPool.java:502) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.client.HttpChannelPool.lambda$connect$4(HttpChannelPool.java:388) ~[armeria-1.3.0.jar!/:?] at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:578) ~[netty-common-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:552) ~[netty-common-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:491) ~[netty-common-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616) ~[netty-common-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605) ~[netty-common-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:104) ~[netty-common-4.1.54.Final.jar!/:4.1.54.Final] at com.linecorp.armeria.client.HttpSessionHandler.tryCompleteSessionPromise(HttpSessionHandler.java:379) ~[armeria-1.3.0.jar!/:?] at com.linecorp.armeria.client.HttpSessionHandler.userEventTriggered(HttpSessionHandler.java:345) ~[armeria-1.3.0.jar!/:?] at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at com.linecorp.armeria.client.Http1ResponseDecoder.userEventTriggered(Http1ResponseDecoder.java:287) ~[armeria-1.3.0.jar!/:?] at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireUserEventTriggered(CombinedChannelDuplexHandler.java:430) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.ChannelInboundHandlerAdapter.userEventTriggered(ChannelInboundHandlerAdapter.java:117) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.handler.codec.ByteToMessageDecoder.userEventTriggered(ByteToMessageDecoder.java:365) ~[netty-codec-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.handler.codec.http.HttpObjectDecoder.userEventTriggered(HttpObjectDecoder.java:491) ~[netty-codec-http-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.CombinedChannelDuplexHandler.userEventTriggered(CombinedChannelDuplexHandler.java:241) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.handler.logging.LoggingHandler.userEventTriggered(LoggingHandler.java:213) ~[netty-handler-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.AbstractChannelHandlerContext.fireUserEventTriggered(AbstractChannelHandlerContext.java:324) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.DefaultChannelPipeline$HeadContext.userEventTriggered(DefaultChannelPipeline.java:1428) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:346) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.AbstractChannelHandlerContext.invokeUserEventTriggered(AbstractChannelHandlerContext.java:332) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.DefaultChannelPipeline.fireUserEventTriggered(DefaultChannelPipeline.java:913) ~[netty-transport-4.1.54.Final.jar!/:4.1.54.Final] at com.linecorp.armeria.client.HttpClientPipelineConfigurator.lambda$finishSuccessfully$0(HttpClientPipelineConfigurator.java:348) ~[armeria-1.3.0.jar!/:?] at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) ~[netty-common-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) ~[netty-common-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) ~[netty-transport-native-epoll-4.1.54.Final-linux-x86_64.jar!/:4.1.54.Final] at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) ~[netty-common-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.54.Final.jar!/:4.1.54.Final] at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.54.Final.jar!/:4.1.54.Final] at java.lang.Thread.run(Thread.java:748) [?:1.8.0_291] `

[gcdd1993]

mark

[jcchavezs]

Could you all please specify the zipkin server version for the bug?

On Thu, Dec 23, 2021, 10:27 观沧海 @.***> wrote:

mark

— Reply to this email directly, view it on GitHub https://github.com/openzipkin/zipkin/issues/3352#issuecomment-1000163009, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXOYAVN5ORGBSQ4C5QB62DUSLTOXANCNFSM44F4GESA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

[gcdd1993]

@jcchavezs hi

my zipkin version 2.23.9

docker-compose.yml

version: '3.7'

services:
  zipkin:
    image: openzipkin/zipkin
    container_name: zipkin
    ports:
      - "9411:9411"
    restart: always
    # https://github.com/openzipkin/zipkin/tree/master/zipkin-server#configuration
    environment:
      - STORAGE_TYPE=MySQL
      - MYSQL_HOST=192.168.2.19
      - MYSQL_TCP_PORT=3306
      - MYSQL_DB=zipkin
      - MYSQL_USER=root
      - MYSQL_PASS=123456

and the exception stack trace

zipkin    |     |__  /_ _|  _ \| |/ /_ _| \ | |
zipkin    |       / / | || |_) | ' / | ||  \| |
zipkin    |      / /_ | ||  __/| . \ | || |\  |
zipkin    |     |____|___|_|   |_|\_\___|_| \_|
zipkin    | 
zipkin    | :: version 2.23.9 :: commit d6b1cc3 ::
zipkin    | 
zipkin    | 2021-12-23 09:32:01.264  INFO [/] 1 --- [oss-http-*:9411] c.l.a.s.Server                           : Serving HTTP at /0.0.0.0:9411 - http://127.0.0.1:9411/
zipkin    | 2021-12-23 09:33:37.757  WARN [/] 1 --- [orker-epoll-2-7] z.s.i.BodyIsExceptionMessage             : Unexpected error handling request.
zipkin    | 
zipkin    | com.linecorp.armeria.server.RequestTimeoutException: null
zipkin    | 	at com.linecorp.armeria.server.RequestTimeoutException.get(RequestTimeoutException.java:36) ~[armeria-1.13.3.jar:?]
zipkin    | 	at com.linecorp.armeria.internal.common.CancellationScheduler.invokeTask(CancellationScheduler.java:467) ~[armeria-1.13.3.jar:?]
zipkin    | 	at com.linecorp.armeria.internal.common.CancellationScheduler.lambda$setTimeoutNanosFromNow0$13(CancellationScheduler.java:293) ~[armeria-1.13.3.jar:?]
zipkin    | 	at com.linecorp.armeria.common.RequestContext.lambda$makeContextAware$3(RequestContext.java:547) ~[armeria-1.13.3.jar:?]
zipkin    | 	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) [netty-common-4.1.69.Final.jar:4.1.69.Final]
zipkin    | 	at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) [netty-common-4.1.69.Final.jar:4.1.69.Final]
zipkin    | 	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-common-4.1.69.Final.jar:4.1.69.Final]
zipkin    | 	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) [netty-common-4.1.69.Final.jar:4.1.69.Final]
zipkin    | 	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) [netty-transport-native-epoll-4.1.69.Final-linux-x86_64.jar:4.1.69.Final]
zipkin    | 	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.69.Final.jar:4.1.69.Final]
zipkin    | 	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.69.Final.jar:4.1.69.Final]
zipkin    | 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.69.Final.jar:4.1.69.Final]
zipkin    | 	at java.lang.Thread.run(Unknown Source) [?:?]
zipkin    | 
zipkin    | 2021-12-23 09:35:27.387  WARN [/] 1 --- [orker-epoll-2-4] z.s.i.BodyIsExceptionMessage             : Unexpected error handling request.
zipkin    | 
zipkin    | com.linecorp.armeria.server.RequestTimeoutException: null
zipkin    | 	at com.linecorp.armeria.server.RequestTimeoutException.get(RequestTimeoutException.java:36) ~[armeria-1.13.3.jar:?]
zipkin    | 	at com.linecorp.armeria.internal.common.CancellationScheduler.invokeTask(CancellationScheduler.java:467) ~[armeria-1.13.3.jar:?]
zipkin    | 	at com.linecorp.armeria.internal.common.CancellationScheduler.lambda$setTimeoutNanosFromNow0$13(CancellationScheduler.java:293) ~[armeria-1.13.3.jar:?]
zipkin    | 	at com.linecorp.armeria.common.RequestContext.lambda$makeContextAware$3(RequestContext.java:547) ~[armeria-1.13.3.jar:?]
zipkin    | 	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) [netty-common-4.1.69.Final.jar:4.1.69.Final]
zipkin    | 	at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) [netty-common-4.1.69.Final.jar:4.1.69.Final]
zipkin    | 	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-common-4.1.69.Final.jar:4.1.69.Final]
zipkin    | 	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) [netty-common-4.1.69.Final.jar:4.1.69.Final]
zipkin    | 	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) [netty-transport-native-epoll-4.1.69.Final-linux-x86_64.jar:4.1.69.Final]
zipkin    | 	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.69.Final.jar:4.1.69.Final]
zipkin    | 	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.69.Final.jar:4.1.69.Final]
zipkin    | 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.69.Final.jar:4.1.69.Final]
zipkin    | 	at java.lang.Thread.run(Unknown Source) [?:?]

by the way, mysql database count

• zipkin_spans: 5221670
• zipkin_annotations: 40168843
• zipkin_dependencies: 0

-> Count rows in zipkin_spans  (actual time=21363.395..21363.395 rows=1 loops=1)

[jcchavezs]

@gcdd1993 I think your issue is different. It seems you have a good load of spans and I am not sure if MySQL can support it. Remember MySQL is not meant for production environments. I think your issue is different from the one described originally.

@msmsimondean could you please specify your Zipkin version?