apache
Thrift Error occurred during processing of message. org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset [Bug]
Search before asking
- [X] I searched in the issues and found nothing similar.
Version
I am running IOTDB 0.13.3 version on a centos 7 machine.
Describe the bug and provide the minimal reproduce step
I have inserted approx 7000 timeseriesinto the databases and trying to get all of them based on a filter time condition. I have increased themax_deduplicated_path_numvalue to 10000. When I am trying to get thetimeseriesI am getting the below error inlogs/log_all.log`
2023-01-25 15:08:21,520 [pool-19-IoTDB-RPC-Client-6] ERROR o.a.t.s.TThreadPoolServer$WorkerProcess:258 - Thrift Error occurred during processing of message. org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:196) at org.apache.iotdb.rpc.TElasticFramedTransport.flush(TElasticFramedTransport.java:135) at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:60) at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:38) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:750) Caused by: java.net.SocketException: Connection reset at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:115) at java.net.SocketOutputStream.write(SocketOutputStream.java:155) at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122) at org.apache.thrift.transport.TIOStreamTransport.write(TIOStreamTransport.java:194) ... 7 common frames omitted 2023-01-25 15:08:21,520 [pool-19-IoTDB-RPC-Client-6] INFO o.a.i.d.s.b.ServiceProvider:232 - Session-6 is closing
I am getting the below error in the client:
15:07:53.708 [WorkerThread0] DEBUG org.apache.iotdb.rpc.AutoResizingBuffer - org.apache.iotdb.rpc.AutoResizingBuffer@9718f40 expand from 65536 t$ 98304, request: 65539 15:07:53.712 [WorkerThread0] DEBUG org.apache.iotdb.rpc.AutoResizingBuffer - org.apache.iotdb.rpc.AutoResizingBuffer@9718f40 expand from 98304 t$ 147456, request: 98306 15:07:53.713 [WorkerThread0] DEBUG org.apache.iotdb.rpc.AutoResizingBuffer - org.apache.iotdb.rpc.AutoResizingBuffer@9718f40 expand from 147456 $ o 221184, request: 147478 15:07:53.715 [WorkerThread0] DEBUG org.apache.iotdb.rpc.AutoResizingBuffer - org.apache.iotdb.rpc.AutoResizingBuffer@9718f40 expand from 221184 $ o 331776, request: 221188 Wed Jan 25 15:08:24 IST 2023 - Fail to reconnect to server. Please check server status
I am using Java Native API to query the data. Is there anything I am missing? Can anyone help me in fixing this?
What did you expect to see?
I expect the program to run without any error and gives me the result of the query
What did you see instead?
I got exception in the server side.
Anything else?
No response
Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
Hi, this is your first issue in IoTDB project. Thanks for your report. Welcome to join the community!
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
@HTHou Fine i read the issue you pointed out and they said that the changes took place back in V0.17 we are now in V0.19 and on my IoTDB cluster i have plenty of those Error.
To give an example:
2024-02-02 10:45:26,139 [pool-52-IoTDB-ClientRPC-Processor-494] ERROR o.a.t.s.TThreadPoolServer$WorkerProcess:258 - Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:178)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
at org.apache.iotdb.rpc.TElasticFramedTransport.readFrame(TElasticFramedTransport.java:113)
at org.apache.iotdb.rpc.TElasticFramedTransport.read(TElasticFramedTransport.java:108)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:463)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:361)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:244)
at org.apache.iotdb.db.protocol.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:50)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: java.net.SocketException: Connection reset
at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:313)
at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:340)
at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:789)
at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1025)
at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:255)
at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:310)
at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:382)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:361)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:176)
... 12 common frames omitted
2024-02-02 10:45:27,181 [pool-52-IoTDB-ClientRPC-Processor-494] ERROR o.a.t.s.TThreadPoolServer$WorkerProcess:258 - Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:178)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
at org.apache.iotdb.rpc.TElasticFramedTransport.readFrame(TElasticFramedTransport.java:113)
at org.apache.iotdb.rpc.TElasticFramedTransport.read(TElasticFramedTransport.java:108)
at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109)
at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:463)
at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:361)
at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:244)
at org.apache.iotdb.db.protocol.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:50)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1589)
Caused by: java.net.SocketException: Connection reset
at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:313)
at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:340)
at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:789)
at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1025)
at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:255)
at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:310)
at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:382)
at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:361)
at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:176)
... 12 common frames omitted
Additional Info:
- IoTDB Version: 1.3.0
- IoTDB BuildInfo: fe787ca
- I'm using Telegraf to send data to the Cluster paired with HAProxy
- Cluster details: 3D 3C
好吧，我阅读了您指出的问题，他们说更改发生在 V0.17 中，我们现在在 V0.19 中，在我的 IoTDB 集群上，我有很多这样的错误。
举个例子：
2024-02-02 10:45:26,139 [pool-52-IoTDB-ClientRPC-Processor-494] ERROR o.a.t.s.TThreadPoolServer$WorkerProcess:258 - Thrift Error occurred during processing of message. org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:178) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109) at org.apache.iotdb.rpc.TElasticFramedTransport.readFrame(TElasticFramedTransport.java:113) at org.apache.iotdb.rpc.TElasticFramedTransport.read(TElasticFramedTransport.java:108) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:463) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:361) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:244) at org.apache.iotdb.db.protocol.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:50) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1589) Caused by: java.net.SocketException: Connection reset at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:313) at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:340) at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:789) at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1025) at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:255) at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:310) at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:382) at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:361) at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:176) ... 12 common frames omitted 2024-02-02 10:45:27,181 [pool-52-IoTDB-ClientRPC-Processor-494] ERROR o.a.t.s.TThreadPoolServer$WorkerProcess:258 - Thrift Error occurred during processing of message. org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:178) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109) at org.apache.iotdb.rpc.TElasticFramedTransport.readFrame(TElasticFramedTransport.java:113) at org.apache.iotdb.rpc.TElasticFramedTransport.read(TElasticFramedTransport.java:108) at org.apache.thrift.transport.TTransport.readAll(TTransport.java:109) at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:463) at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:361) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:244) at org.apache.iotdb.db.protocol.thrift.ProcessorWithMetrics.process(ProcessorWithMetrics.java:50) at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:248) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1589) Caused by: java.net.SocketException: Connection reset at java.base/sun.nio.ch.NioSocketImpl.implRead(NioSocketImpl.java:313) at java.base/sun.nio.ch.NioSocketImpl.read(NioSocketImpl.java:340) at java.base/sun.nio.ch.NioSocketImpl$1.read(NioSocketImpl.java:789) at java.base/java.net.Socket$SocketInputStream.read(Socket.java:1025) at java.base/java.io.BufferedInputStream.fill(BufferedInputStream.java:255) at java.base/java.io.BufferedInputStream.read1(BufferedInputStream.java:310) at java.base/java.io.BufferedInputStream.implRead(BufferedInputStream.java:382) at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:361) at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:176) ... 12 common frames omitted附加信息：
- IoTDB版本：1.3.0
- IoTDB 构建信息： fe787ca
- 我正在使用 Telegraf 将数据发送到与 HAProxy 配对的集群
- 集群详细信息：3D 3C
一样，你解决了吗？都是thrift报错了后，无法写入数据了，cli也连不上，workbench连不上。
@zhqingyong Hi, I have solved this issue some times ago same as other open issues...... the community and the developer are not really helpfull. Anyway to fix your issue you must set the backend check in HAproxis as the following lines:
for port 18080 you can set up health checking as basic for port 6667 you MUST set up health checking as none
the error "org.apache.thrift.transport.TTransportException: java.net.SocketException: Connection reset" means that the connection is closed brutally (The HA Proxy basic health check on port 6667 is the cause) changing the settings to none will fix your issue
I really hope the developers will be more active in the future. 🙏