Runtime Exception when executing a query twice in a short duration

Open TessaIO opened this issue 6 months ago • 1 comments

Historical crashes with Runtime exception when querying

Affected Version

32.0.0

Description

We have a cluster of 20 historicals with 4 brokers. There are some queries when being executed in a short duration twice they just error out with the following issue:

2025-05-27T14:22:08,950 WARN [qtp92221031-168[groupBy_[datasource]_c532831b-3598-4fa6-a384-aff92a767770]] org.apache.druid.query.groupby.GroupByQueryResources - 1 merging query runner merge buffers are not returned yet
2025-05-27T14:22:08,952 WARN [qtp92221031-168[groupBy_[datasource]_c532831b-3598-4fa6-a384-aff92a767770]] org.apache.druid.server.QueryLifecycle - Exception while processing queryId [c532831b-3598-4fa6-a384-aff92a767770]
DruidException context: {host=null, errorClass=java.lang.ArrayIndexOutOfBoundsException, legacyErrorCode=Unknown exception}
org.apache.druid.error.DruidException: Index 180 out of bounds for length 129
	at org.apache.druid.error.QueryExceptionCompat.makeException(QueryExceptionCompat.java:50) ~[druid-processing-32.0.0.jar:32.0.0]
	at org.apache.druid.error.DruidException.fromFailure(DruidException.java:157) ~[druid-processing-32.0.0.jar:32.0.0]
	at org.apache.druid.server.QueryResultPusher.handleQueryException(QueryResultPusher.java:229) ~[druid-server-32.0.0.jar:32.0.0]
	at org.apache.druid.server.QueryResultPusher.push(QueryResultPusher.java:195) ~[druid-server-32.0.0.jar:32.0.0]
	at org.apache.druid.server.QueryResource.doPost(QueryResource.java:222) ~[druid-server-32.0.0.jar:32.0.0]
	at jdk.internal.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) ~[?:?]
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
	at java.base/java.lang.reflect.Method.invoke(Method.java:569) ~[?:?]
	at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) ~[jersey-server-1.19.4.jar:1.19.4]
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) ~[jersey-servlet-1.19.4.jar:1.19.4]
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) ~[jersey-servlet-1.19.4.jar:1.19.4]
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) ~[jersey-servlet-1.19.4.jar:1.19.4]
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
	at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290) ~[guice-servlet-5.1.0.jar:?]
	at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280) ~[guice-servlet-5.1.0.jar:?]
	at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184) ~[guice-servlet-5.1.0.jar:?]
	at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89) ~[guice-servlet-5.1.0.jar:?]
	at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) ~[guice-servlet-5.1.0.jar:?]
	at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121) ~[guice-servlet-5.1.0.jar:?]
	at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133) ~[guice-servlet-5.1.0.jar:?]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.apache.druid.server.security.PreResponseAuthorizationCheckFilter.doFilter(PreResponseAuthorizationCheckFilter.java:84) ~[druid-server-32.0.0.jar:32.0.0]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.apache.druid.server.initialization.jetty.StandardResponseHeaderFilterHolder$StandardResponseHeaderFilter.doFilter(StandardResponseHeaderFilterHolder.java:164) ~[druid-server-32.0.0.jar:32.0.0]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.apache.druid.server.security.AllowHttpMethodsResourceFilter.doFilter(AllowHttpMethodsResourceFilter.java:78) ~[druid-server-32.0.0.jar:32.0.0]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.apache.druid.server.security.AllowOptionsResourceFilter.doFilter(AllowOptionsResourceFilter.java:74) ~[druid-server-32.0.0.jar:32.0.0]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.apache.druid.security.basic.authentication.BasicHTTPAuthenticator$BasicHTTPAuthenticationFilter.doFilter(BasicHTTPAuthenticator.java:220) ~[?:?]
	at org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59) ~[druid-server-32.0.0.jar:32.0.0]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.apache.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:77) ~[druid-server-32.0.0.jar:32.0.0]
	at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) [jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) [jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) [jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
	at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
	at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
Caused by: org.apache.druid.query.QueryInterruptedException: Index 180 out of bounds for length 129
	... 79 more
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 180 out of bounds for length 129
	at it.unimi.dsi.fastutil.ints.Int2ObjectOpenHashMap.get(Int2ObjectOpenHashMap.java:284) ~[fastutil-core-8.5.4.jar:?]
	at org.apache.druid.query.aggregation.datasketches.theta.SketchBufferAggregatorHelper.get(SketchBufferAggregatorHelper.java:66) ~[?:?]
	at org.apache.druid.query.aggregation.datasketches.theta.SketchBufferAggregator.get(SketchBufferAggregator.java:64) ~[?:?]
	at org.apache.druid.query.groupby.epinephelinae.StreamingMergeSortedGrouper$1.next(StreamingMergeSortedGrouper.java:438) ~[druid-processing-32.0.0.jar:32.0.0]
	at org.apache.druid.query.groupby.epinephelinae.StreamingMergeSortedGrouper$1.next(StreamingMergeSortedGrouper.java:387) ~[druid-processing-32.0.0.jar:32.0.0]
	at org.apache.druid.java.util.common.CloseableIterators$1.next(CloseableIterators.java:78) ~[druid-processing-32.0.0.jar:32.0.0]
	at org.apache.druid.query.groupby.epinephelinae.CloseableGrouperIterator.next(CloseableGrouperIterator.java:53) ~[druid-processing-32.0.0.jar:32.0.0]
	at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:44) ~[druid-processing-32.0.0.jar:32.0.0]
	at org.apache.druid.common.guava.CombiningSequence.accumulate(CombiningSequence.java:62) ~[druid-processing-32.0.0.jar:32.0.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-processing-32.0.0.jar:32.0.0]
	at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-processing-32.0.0.jar:32.0.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-processing-32.0.0.jar:32.0.0]
	at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-processing-32.0.0.jar:32.0.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-processing-32.0.0.jar:32.0.0]
	at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:77) ~[druid-processing-32.0.0.jar:32.0.0]
	at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-processing-32.0.0.jar:32.0.0]
	at org.apache.druid.server.QueryResultPusher.push(QueryResultPusher.java:166) ~[druid-server-32.0.0.jar:32.0.0]
	... 78 more
	Suppressed: java.lang.RuntimeException: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
		at org.apache.druid.query.groupby.epinephelinae.ParallelCombiner.checkCombineFutures(ParallelCombiner.java:204) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.query.groupby.epinephelinae.ParallelCombiner.lambda$combine$0(ParallelCombiner.java:174) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.java.util.common.io.Closer.close(Closer.java:170) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.java.util.common.CloseableIterators$1.close(CloseableIterators.java:86) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.java.util.common.io.Closer.close(Closer.java:170) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.query.groupby.epinephelinae.CloseableGrouperIterator.close(CloseableGrouperIterator.java:72) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunner$1.cleanup(GroupByMergingQueryRunner.java:319) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.query.groupby.epinephelinae.GroupByMergingQueryRunner$1.cleanup(GroupByMergingQueryRunner.java:183) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.java.util.common.guava.BaseSequence.accumulate(BaseSequence.java:49) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.common.guava.CombiningSequence.accumulate(CombiningSequence.java:62) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.java.util.common.guava.SequenceWrapper.wrap(SequenceWrapper.java:55) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.java.util.common.guava.MappedSequence.accumulate(MappedSequence.java:43) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.java.util.common.guava.WrappingSequence$1.get(WrappingSequence.java:50) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.query.CPUTimeMetricQueryRunner$1.wrap(CPUTimeMetricQueryRunner.java:77) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.java.util.common.guava.WrappingSequence.accumulate(WrappingSequence.java:45) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.server.QueryResultPusher.push(QueryResultPusher.java:166) ~[druid-server-32.0.0.jar:32.0.0]
		at org.apache.druid.server.QueryResource.doPost(QueryResource.java:222) ~[druid-server-32.0.0.jar:32.0.0]
		at jdk.internal.reflect.GeneratedMethodAccessor80.invoke(Unknown Source) ~[?:?]
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
		at java.base/java.lang.reflect.Method.invoke(Method.java:569) ~[?:?]
		at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60) ~[jersey-server-1.19.4.jar:1.19.4]
		at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205) ~[jersey-server-1.19.4.jar:1.19.4]
		at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75) ~[jersey-server-1.19.4.jar:1.19.4]
		at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302) ~[jersey-server-1.19.4.jar:1.19.4]
		at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108) ~[jersey-server-1.19.4.jar:1.19.4]
		at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147) ~[jersey-server-1.19.4.jar:1.19.4]
		at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84) ~[jersey-server-1.19.4.jar:1.19.4]
		at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1542) ~[jersey-server-1.19.4.jar:1.19.4]
		at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1473) ~[jersey-server-1.19.4.jar:1.19.4]
		at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419) ~[jersey-server-1.19.4.jar:1.19.4]
		at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409) ~[jersey-server-1.19.4.jar:1.19.4]
		at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409) ~[jersey-servlet-1.19.4.jar:1.19.4]
		at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558) ~[jersey-servlet-1.19.4.jar:1.19.4]
		at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733) ~[jersey-servlet-1.19.4.jar:1.19.4]
		at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) ~[javax.servlet-api-3.1.0.jar:3.1.0]
		at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290) ~[guice-servlet-5.1.0.jar:?]
		at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280) ~[guice-servlet-5.1.0.jar:?]
		at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184) ~[guice-servlet-5.1.0.jar:?]
		at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89) ~[guice-servlet-5.1.0.jar:?]
		at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) ~[guice-servlet-5.1.0.jar:?]
		at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121) ~[guice-servlet-5.1.0.jar:?]
		at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133) ~[guice-servlet-5.1.0.jar:?]
		at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.apache.druid.server.security.PreResponseAuthorizationCheckFilter.doFilter(PreResponseAuthorizationCheckFilter.java:84) ~[druid-server-32.0.0.jar:32.0.0]
		at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.apache.druid.server.initialization.jetty.StandardResponseHeaderFilterHolder$StandardResponseHeaderFilter.doFilter(StandardResponseHeaderFilterHolder.java:164) ~[druid-server-32.0.0.jar:32.0.0]
		at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.apache.druid.server.security.AllowHttpMethodsResourceFilter.doFilter(AllowHttpMethodsResourceFilter.java:78) ~[druid-server-32.0.0.jar:32.0.0]
		at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.apache.druid.server.security.AllowOptionsResourceFilter.doFilter(AllowOptionsResourceFilter.java:74) ~[druid-server-32.0.0.jar:32.0.0]
		at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.apache.druid.security.basic.authentication.BasicHTTPAuthenticator$BasicHTTPAuthenticationFilter.doFilter(BasicHTTPAuthenticator.java:220) ~[?:?]
		at org.apache.druid.server.security.AuthenticationWrappingFilter.doFilter(AuthenticationWrappingFilter.java:59) ~[druid-server-32.0.0.jar:32.0.0]
		at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.apache.druid.server.security.SecuritySanityCheckFilter.doFilter(SecuritySanityCheckFilter.java:77) ~[druid-server-32.0.0.jar:32.0.0]
		at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1626) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:552) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505) ~[jetty-servlet-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:59) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:181) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.server.Server.handle(Server.java:516) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487) ~[jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732) [jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479) [jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277) [jetty-server-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) [jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) [jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) [jetty-io-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338) [jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315) [jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173) [jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131) [jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409) [jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883) [jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
		at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034) [jetty-util-9.4.56.v20240826.jar:9.4.56.v20240826]
		at java.base/java.lang.Thread.run(Thread.java:840) [?:?]
	Caused by: java.util.concurrent.ExecutionException: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
		at java.base/java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:?]
		at java.base/java.util.concurrent.FutureTask.get(FutureTask.java:191) ~[?:?]
		at org.apache.druid.query.PrioritizedListenableFutureTask.get(PrioritizedExecutorService.java:283) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.query.groupby.epinephelinae.ParallelCombiner.checkCombineFutures(ParallelCombiner.java:197) ~[druid-processing-32.0.0.jar:32.0.0]
		... 95 more
	Caused by: java.lang.IndexOutOfBoundsException: Index 0 out of bounds for length 0
		at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) ~[?:?]
		at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) ~[?:?]
		at java.base/jdk.internal.util.Preconditions.checkIndex(Preconditions.java:266) ~[?:?]
		at java.base/java.util.Objects.checkIndex(Objects.java:361) ~[?:?]
		at java.base/java.util.ArrayList.get(ArrayList.java:427) ~[?:?]
		at org.apache.druid.query.groupby.epinephelinae.RowBasedGrouperHelper$RowBasedKeySerde$AbstractStringRowBasedKeySerdeHelper.getFromByteBuffer(RowBasedGrouperHelper.java:1896) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.query.groupby.epinephelinae.RowBasedGrouperHelper$RowBasedKeySerde.readFromByteBuffer(RowBasedGrouperHelper.java:1339) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.query.groupby.epinephelinae.RowBasedGrouperHelper$RowBasedKeySerde.readFromByteBuffer(RowBasedGrouperHelper.java:1156) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.query.groupby.epinephelinae.AbstractBufferHashGrouper.populateBucketEntryForOffset(AbstractBufferHashGrouper.java:185) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.query.groupby.epinephelinae.BufferHashGrouper$2.next(BufferHashGrouper.java:234) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.query.groupby.epinephelinae.BufferHashGrouper$2.next(BufferHashGrouper.java:216) ~[druid-processing-32.0.0.jar:32.0.0]
		at com.google.common.collect.Iterators$PeekingImpl.peek(Iterators.java:1217) ~[guava-32.0.1-jre.jar:?]
		at org.apache.druid.java.util.common.guava.MergeIterator.lambda$new$0(MergeIterator.java:53) ~[druid-processing-32.0.0.jar:32.0.0]
		at java.base/java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:660) ~[?:?]
		at java.base/java.util.PriorityQueue.siftUp(PriorityQueue.java:637) ~[?:?]
		at java.base/java.util.PriorityQueue.offer(PriorityQueue.java:330) ~[?:?]
		at java.base/java.util.PriorityQueue.add(PriorityQueue.java:311) ~[?:?]
		at org.apache.druid.java.util.common.guava.MergeIterator.next(MergeIterator.java:81) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.java.util.common.CloseableIterators$1.next(CloseableIterators.java:78) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.java.util.common.CloseableIterators$1.next(CloseableIterators.java:78) ~[druid-processing-32.0.0.jar:32.0.0]
		at com.google.common.collect.Iterators$PeekingImpl.peek(Iterators.java:1217) ~[guava-32.0.1-jre.jar:?]
		at org.apache.druid.java.util.common.guava.MergeIterator.lambda$new$0(MergeIterator.java:53) ~[druid-processing-32.0.0.jar:32.0.0]
		at java.base/java.util.PriorityQueue.siftUpUsingComparator(PriorityQueue.java:660) ~[?:?]
		at java.base/java.util.PriorityQueue.siftUp(PriorityQueue.java:637) ~[?:?]
		at java.base/java.util.PriorityQueue.offer(PriorityQueue.java:330) ~[?:?]
		at java.base/java.util.PriorityQueue.add(PriorityQueue.java:311) ~[?:?]
		at org.apache.druid.java.util.common.guava.MergeIterator.next(MergeIterator.java:81) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.java.util.common.CloseableIterators$1.next(CloseableIterators.java:78) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.query.groupby.epinephelinae.ParallelCombiner$2.call(ParallelCombiner.java:414) ~[druid-processing-32.0.0.jar:32.0.0]
		at org.apache.druid.query.groupby.epinephelinae.ParallelCombiner$2.call(ParallelCombiner.java:400) ~[druid-processing-32.0.0.jar:32.0.0]
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
		at org.apache.druid.query.PrioritizedListenableFutureTask.run(PrioritizedExecutorService.java:259) ~[druid-processing-32.0.0.jar:32.0.0]
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
		... 1 more

This is pretty flaky, and sometimes it happens immediately, sometimes after two consecutive runs, and sometimes even after 5 to 6 runs. It should be noted that the query is a groupBy query that goes over 6 datasources whose sizes are in the order of few GBs. Also I'm disabling the cache when I have the issue.

May 27 '25 14:05 TessaIO

The deepest cause of the "Index 180 out of bounds for length 129" error is the stack trace below, which I think can only happen if the unionMap is corrupted somehow, which I think can only happen if a SketchBufferAggregator is being used by two threads at once. That would be a violation of the contract of BufferAggregator, which is not thread safe.

@TessaIO if you disable parallel combining (numParallelCombineThreads: 1) do you still see this problem? I wonder if it is some issue with that feature. I think 1 (i.e. no parallel combining) is the default.

Caused by: java.lang.ArrayIndexOutOfBoundsException: Index 180 out of bounds for length 129
	at it.unimi.dsi.fastutil.ints.Int2ObjectOpenHashMap.get(Int2ObjectOpenHashMap.java:284) ~[fastutil-core-8.5.4.jar:?]
	at org.apache.druid.query.aggregation.datasketches.theta.SketchBufferAggregatorHelper.get(SketchBufferAggregatorHelper.java:66) ~[?:?]
	at org.apache.druid.query.aggregation.datasketches.theta.SketchBufferAggregator.get(SketchBufferAggregator.java:64) ~[?:?]
	at org.apache.druid.query.groupby.epinephelinae.StreamingMergeSortedGrouper$1.next(StreamingMergeSortedGrouper.java:438) ~[druid-processing-32.0.0.jar:32.0.0]
	at org.apache.druid.query.groupby.epinephelinae.StreamingMergeSortedGrouper$1.next(StreamingMergeSortedGrouper.java:387) ~[druid-processing-32.0.0.jar:32.0.0]
	at org.apache.druid.java.util.common.CloseableIterators$1.next(CloseableIterators.java:78) ~[druid-processing-32.0.0.jar:32.0.0]
	at org.apache.druid.query.groupby.epinephelinae.CloseableGrouperIterator.next(CloseableGrouperIterator.java:53) ~[druid-processing-32.0.0.jar:32.0.0]

May 31 '25 00:05 gianm