DTStack
mysql同步到hdfs报错
Search before asking
- [X] I had searched in the issues and found no similar issues.
What happened
2022-08-23 15:49:52.052 WARN 51 -- [dfswriter (1/1)] com.dtstack.flinkx.rdb.util.DbUtil : Close resultSet error: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link failure
The last packet successfully received from the server was 7 milliseconds ago. The last packet sent successfully to the server was 1,107 milliseconds ago. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:990) at com.mysql.jdbc.MysqlIO.clearInputStream(MysqlIO.java:903) at com.mysql.jdbc.RowDataDynamic.close(RowDataDynamic.java:172) at com.mysql.jdbc.ResultSetImpl.realClose(ResultSetImpl.java:6679) at com.mysql.jdbc.ResultSetImpl.close(ResultSetImpl.java:851) at com.dtstack.flinkx.rdb.util.DbUtil.closeDbResources(DbUtil.java:184) at com.dtstack.flinkx.rdb.inputformat.JdbcInputFormat.closeInternal(JdbcInputFormat.java:278) at com.dtstack.flinkx.inputformat.BaseRichInputFormat.close(BaseRichInputFormat.java:338) at com.dtstack.flinkx.streaming.api.functions.source.DtInputFormatSourceFunction.run(DtInputFormatSourceFunction.java:154) at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100) at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63) at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:200) Caused by: javax.net.ssl.SSLException: Connection has been shutdown: javax.net.ssl.SSLException: java.net.SocketException: Connection reset at sun.security.ssl.SSLSocketImpl.checkEOF(SSLSocketImpl.java:1554) at sun.security.ssl.AppInputStream.available(AppInputStream.java:60) at com.mysql.jdbc.MysqlIO.clearInputStream(MysqlIO.java:899) ... 10 more Caused by: javax.net.ssl.SSLException: java.net.SocketException: Connection reset at sun.security.ssl.Alerts.getSSLException(Alerts.java:214) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1967) at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1924) at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1888) at sun.security.ssl.SSLSocketImpl.handleException(SSLSocketImpl.java:1833) at sun.security.ssl.AppInputStream.read(AppInputStream.java:116) at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3011) at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3522) at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3462) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3903) at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:871) at com.mysql.jdbc.MysqlIO.nextRow(M ysqlIO.java:1999) at com.mysql.jdbc.RowDataDynamic.nextRecord(RowDataDynamic.java:374) at com.mysql.jdbc.RowDataDynamic.next(RowDataDynamic.java:354) at com.mysql.jdbc.ResultSetImpl.next(ResultSetImpl.java:6312) at com.dtstack.flinkx.rdb.inputformat.JdbcInputFormat.nextRecordInternal(JdbcInputFormat.java:249) at com.dtstack.flinkx.mysql.format.MysqlInputFormat.nextRecordInternal(MysqlInputFormat.java:78) at com.dtstack.flinkx.inputformat.BaseRichInputFormat.nextRecord(BaseRichInputFormat.java:282) at com.dtstack.flinkx.inputformat.BaseRichInputFormat.nextRecord(BaseRichInputFormat.java:66) at com.dtstack.flinkx.streaming.api.functions.source.DtInputFormatSourceFunction.run(DtInputFormatSourceFunction.java:137) ... 3 more Caused by: java.net.SocketException: Connection reset at java.net.SocketInputStream.read(SocketInputStream.java:210) at java.net.SocketInputStream.read(SocketInputStream.java:141) at sun.security.ssl.InputRecord.readFully(InputRecord.java:465) at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:593) at sun.security.ssl.InputRecord.read(InputRecord.java:532) at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:990) at sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:948) at sun.security.ssl.AppInputStream.read(AppInputStream.java:105) ... 17 more
2022-08-23 15:49:52.054 WARN 51 -- [dfswriter (1/1)] com.dtstack.flinkx.rdb.util.DbUtil : commit error:com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Communications link failure during commit(). Transaction resolution unknown. at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at com.mysql.jdbc.Util.handleNewInstance(Util.java:425) at com.mysql.jdbc.Util.getInstance(Util.java:408) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:919) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:898) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:887) at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:861) at com.mysql.jdbc.ConnectionImpl.commit(ConnectionImpl.java:1551) at com.dtstack.flinkx.rdb.util.DbUtil.commit(DbUtil.java:220) at com.dtstack.flinkx.rdb.util.DbUtil.closeDbResources(DbUtil.java:201) at com.dtstack.flinkx.rdb.inputformat.JdbcInputFormat.closeInternal(JdbcInputFormat.java:278) at com.dtstack.flinkx.inputformat.BaseRichInputFormat.close(BaseRichInputFormat.java:338) at com.dtstack.flinkx.streaming.api.functions.source.DtInputFormatSourceFunction.run(DtInputFormatSourceFunction.java:154) at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100) at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63) at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:200)
What you expected to happen
1
How to reproduce
1
Anything else
1
Version
1.10_release
Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
@momisabuilder 看你这个日志信息是mysql connection 抛出的异常信息；看信息是环境连接问题。你这边任务重跑了多次也会有这个情况吗
mysql能连上。
你这边任务重跑了多次也会有这个情况吗？ 是的，跑了很多次，同样的错误
------------------ 原始邮件 ------------------ 发件人: @.>; 发送时间: 2022年8月23日(星期二) 下午5:11 收件人: @.>; 抄送: "烦不烦 @.>; @.>; 主题: Re: [DTStack/chunjun] mysql同步到hdfs报错 (Issue #1177)
看你这个日志信息是mysql connection 抛出的异常信息；看信息是环境连接问题。你这边任务重跑了多次也会有这个情况吗
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>
5.7.36-log
------------------ 原始邮件 ------------------ 发件人: @.>; 发送时间: 2022年8月24日(星期三) 中午11:07 收件人: @.>; 抄送: "烦不烦 @.>; @.>; 主题: Re: [DTStack/chunjun] mysql同步到hdfs报错 (Issue #1177)
这边都是正常运行的。你这边的mysql是哪个版本的呢
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.Message ID: @.***>
看看你的mysql版本是不是衍生分支Percona版的，这个分支里有个参数 kill_idle_transaction会根据配置的时间会自动杀死事务。
看看你的mysql版本是不是衍生分支Percona版的，这个分支里有个参数 kill_idle_transaction会根据配置的时间会自动杀死事务。
你好，非衍生。