spring-cloud-gateway icon indicating copy to clipboard operation
spring-cloud-gateway copied to clipboard
verified publisher icon spring-cloud

JVM crash when spring-boot version > 2.6.1

Open jalr4ever opened this issue 3 years ago • 0 comments

Bug

Environment: Linux ARM （Detail see below）

05-09 11:39:02.538 [reactor-http-nio-3] ERROR r.n.http.server.HttpServerOperations [error] [315] - [78479837-1, L:/152.20.6.59:8020 - R:/152.20.0.0:29413] Error finishing response. Closing connection
io.netty.handler.codec.DecoderException: java.lang.IllegalStateException: new buffer must have at least one element
 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480)
 Suppressed: reactor.core.publisher.FluxOnAssembly$OnAssemblyException:
Error has been observed at the following site(s):
 *__checkpoint ⇢ org.springframework.cloud.gateway.filter.WeightCalculatorWebFilter [DefaultWebFilterChain]
 *__checkpoint ⇢ org.springframework.boot.actuate.metrics.web.reactive.server.MetricsWebFilter [DefaultWebFilterChain]
 *__checkpoint ⇢ HTTP POST "/cloud-inner/file/download" [ExceptionHandlingWebHandler]
Original Stack Trace:
  at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:480)
  at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
  at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
  at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
  at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
  at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
  at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
  at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
  at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
  at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
  at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
  at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
  at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
  at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
  at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
  at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.IllegalStateException: new buffer must have at least one element
 at io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueue.newBufferPoll(BaseMpscLinkedArrayQueue.java:458)
 at io.netty.util.internal.shaded.org.jctools.queues.BaseMpscLinkedArrayQueue.relaxedPoll(BaseMpscLinkedArrayQueue.java:514)
 at io.netty.util.internal.shaded.org.jctools.queues.MpscChunkedArrayQueue.relaxedPoll(MpscChunkedArrayQueue.java:43)
 at io.netty.util.Recycler$LocalPool.claim(Recycler.java:274)
 at io.netty.util.Recycler.get(Recycler.java:166)
 at io.netty.util.internal.ObjectPool$RecyclerObjectPool.get(ObjectPool.java:84)
 at io.netty.buffer.PooledSlicedByteBuf.newInstance0(PooledSlicedByteBuf.java:52)
 at io.netty.buffer.PooledSlicedByteBuf.newInstance(PooledSlicedByteBuf.java:47)
 at io.netty.buffer.PooledByteBuf.retainedSlice(PooledByteBuf.java:154)
 at io.netty.buffer.AbstractByteBuf.readRetainedSlice(AbstractByteBuf.java:888)
 at io.netty.handler.codec.http.HttpObjectDecoder.decode(HttpObjectDecoder.java:402)
 at io.netty.handler.codec.http.HttpClientCodec$Decoder.decode(HttpClientCodec.java:238)
 at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:510)
 at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449)
 at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279)
 at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
 at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
 at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
 at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
 at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
 at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166)
 at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722)
 at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658)
 at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584)
 at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496)
 at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
 at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 at java.lang.Thread.run(Thread.java:748)
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x0000ffffa1e4c8fc, pid=1, tid=0x0000ffff99abfaa0
#
# JRE version: OpenJDK Runtime Environment (8.0_275-b01) (build 1.8.0_275-b01)
# Java VM: OpenJDK 64-Bit Server VM (25.275-b01 mixed mode linux-aarch64 compressed oops)
# Derivative: IcedTea 3.17.1
# Distribution: Custom build (Tue Feb 16 18:39:49 UTC 2021)
# Problematic frame:
# J 5816 C1 io.netty.channel.ChannelOutboundBuffer.removeBytes(J)V (123 bytes) @ 0x0000ffffa1e4c8fc [0x0000ffffa1e4b480+0x147c]
#
# Core dump written. Default location: //core or core.1
#
# An error report file with more information is saved as:
# /tmp/hs_err_pid1.log
#
# If you would like to submit a bug report, please include
# instructions on how to reproduce the bug and visit:
#   https://icedtea.classpath.org/bugzilla

Introduction

Hi, I use spring cloud gateway as proxy in my project, which deployed in k8s. I manage dependency version by spring-cloud-dependencies and spring-boot-dependencies. Such as:

<dependency>
    <groupId>org.springframework.cloud</groupId>
    <artifactId>spring-cloud-dependencies</artifactId>
    <version>${spring-cloud.version}</version>
    <type>pom</type>
    <scope>import</scope>
</dependency>
<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-dependencies</artifactId>
    <version>${spring-boot.version}</version>
    <type>pom</type>
    <scope>import</scope>
</dependency>

It works well when ${spring-cloud.version} is 2021.0.0 and ${spring-boot.version} is 2.6.1, but problem occurred when I update version:

  • ${spring-cloud.version} to 2021.0.2
  • ${spring-boot.version} to 2.6.3(PS: I have try 2.6.7, there's still the sample problerm)

After I d a download HTTP request, spring cloud gateway seems like could not handle the request and it caused JVM crash?

Any way to fix it?

jalr4ever avatar May 09 '22 07:05 jalr4ever

Well many days have passed now, I try to update the dependencies version like below, which solved this issue:

<properties>
      <spring-boot.version>2.7.3</spring-boot.version>
      <spring-cloud.version>2021.0.4</spring-cloud.version>More
  </properties>
 
<dependencyManagement>
      <dependencies>
 
          <dependency>
              <groupId>org.springframework.cloud</groupId>
              <artifactId>spring-cloud-dependencies</artifactId>
              <version>${spring-cloud.version}</version>
              <type>pom</type>
              <scope>import</scope>
          </dependency>
          <dependency>
              <groupId>org.springframework.boot</groupId>
              <artifactId>spring-boot-dependencies</artifactId>
              <version>${spring-boot.version}</version>
              <type>pom</type>
              <scope>import</scope>
          </dependency>
      </dependencies>
  </dependencyManagement>

More testing shall be, I think?

jalr4ever avatar Sep 19 '22 12:09 jalr4ever