polardbx-sql icon indicating copy to clipboard operation
polardbx-sql copied to clipboard

ERROR jTPCCTData : [144044c5e7001000-8][10.16.9.12:10069][tpcc1]ERR-CODE: [PXC-10001][ERR_X_PROTOCOL_CLIENT] XDataSource to [email protected]:44881 ERR-CODE: [PXC-10003][ERR_X_PROTOCOL_CONNECTION] XClientPool to [email protected]:44881 get XConnection timeout. 5000000000ns

Open wsm12138 opened this issue 2 years ago • 4 comments

VERSION

About PolarDB-X

deployed by pxd pxd version: 0.3.0 polardbx/galaxysql latest "Created": "2022-04-28T11:05:22.088619Z"

About ENV

CentOS Linux release 7.9.2009 (Core) Python 3.6.8 Docker version 20.10.14, build a224086

Problem

After use Benchmarksql build 1,000 warehouses ,start runBenchmark. error occurs

11:00:45,448 [Thread-7] ERROR  jTPCCTData : Unexpected SQLException in NEW_ORDER
11:00:45,448 [Thread-7] ERROR  jTPCCTData : [144044c5e7001000-8][10.16.9.12:10069][tpcc1]ERR-CODE: [PXC-10001][ERR_X_PROTOCOL_CLIENT] XDataSource to [email protected]:44881 ERR-CODE: [PXC-10003][ERR_X_PROTOCOL_CONNECTION] XClientPool to [email protected]:44881 get XConnection timeout. 5000000000ns
java.sql.SQLException: [144044c5e7001000-8][10.16.9.12:10069][tpcc1]ERR-CODE: [PXC-10001][ERR_X_PROTOCOL_CLIENT] XDataSource to [email protected]:44881 ERR-CODE: [PXC-10003][ERR_X_PROTOCOL_CONNECTION] XClientPool to [email protected]:44881 get XConnection timeout. 5000000000ns
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909)
	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527)
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484)
	at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1858)
	at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1966)
	at jTPCCTData.executeNewOrder(jTPCCTData.java:430)
	at jTPCCTData.execute(jTPCCTData.java:95)
	at jTPCCTerminal.executeTransactions(jTPCCTerminal.java:261)
	at jTPCCTerminal.run(jTPCCTerminal.java:88)
	at java.lang.Thread.run(Thread.java:748)

Try to fix

  1. restart cdc,dn,cn,gms of PolarDB-x in docker —— Errors still occur
  2. related #34 ,check variable —— galaxyXProtocol=1
[admin@centos71 drds-server]$ cat conf/server.properties
serverPort=3306
managerPort=3406
rpcPort=9090
charset=utf-8
processors=4
processorHandler=16
processorKillExecutor=128
timerExecutor=8
managerExecutor=256
serverExecutor=1024
idleTimeout=
trustedIps=127.0.0.1
slowSqlTime=1000
maxConnection=20000
allowManagerLogin=1
allowCrossDbQuery=true
galaxyXProtocol=1
metaDbAddr=127.0.0.1:3306
metaDbXprotoPort=13306
metaDbUser=my_polarx
metaDbName=polardbx_meta_db_polardbx
instanceId=polardbx-polardbx
  1. Telnet —— network & port alive image

wsm12138 avatar May 06 '22 03:05 wsm12138

when terminals=50 image

when terminals=200 Errors still occur

target is terminals=500

wsm12138 avatar May 06 '22 03:05 wsm12138

According to the https://www.alibabacloud.com/help/zh/polardb-x/latest/tpc-c-test proved data integrity. image

wsm12138 avatar May 06 '22 03:05 wsm12138

VERSION

About PolarDB-X

deployed by pxd pxd version: 0.3.0 polardbx/galaxysql latest "Created": "2022-04-28T11:05:22.088619Z"

About ENV

CentOS Linux release 7.9.2009 (Core) Python 3.6.8 Docker version 20.10.14, build a224086

Problem

After use Benchmarksql build 1,000 warehouses ,start runBenchmark. error occurs

11:00:45,448 [Thread-7] ERROR  jTPCCTData : Unexpected SQLException in NEW_ORDER
11:00:45,448 [Thread-7] ERROR  jTPCCTData : [144044c5e7001000-8][10.16.9.12:10069][tpcc1]ERR-CODE: [PXC-10001][ERR_X_PROTOCOL_CLIENT] XDataSource to [email protected]:44881 ERR-CODE: [PXC-10003][ERR_X_PROTOCOL_CONNECTION] XClientPool to [email protected]:44881 get XConnection timeout. 5000000000ns
java.sql.SQLException: [144044c5e7001000-8][10.16.9.12:10069][tpcc1]ERR-CODE: [PXC-10001][ERR_X_PROTOCOL_CLIENT] XDataSource to [email protected]:44881 ERR-CODE: [PXC-10003][ERR_X_PROTOCOL_CONNECTION] XClientPool to [email protected]:44881 get XConnection timeout. 5000000000ns
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909)
	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527)
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484)
	at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1858)
	at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1966)
	at jTPCCTData.executeNewOrder(jTPCCTData.java:430)
	at jTPCCTData.execute(jTPCCTData.java:95)
	at jTPCCTerminal.executeTransactions(jTPCCTerminal.java:261)
	at jTPCCTerminal.run(jTPCCTerminal.java:88)
	at java.lang.Thread.run(Thread.java:748)

What about the hardware resources allocated for PolarDB-X? This error (XClientPool to [email protected]:44881 get XConnection timeout) might be caused by a DN crash or hang in the case of high concurrency pressure. You can also check the log file for more information.

F-ca7 avatar May 10 '22 10:05 F-ca7

VERSION

About PolarDB-X

deployed by pxd pxd version: 0.3.0 polardbx/galaxysql latest "Created": "2022-04-28T11:05:22.088619Z"

About ENV

CentOS Linux release 7.9.2009 (Core) Python 3.6.8 Docker version 20.10.14, build a224086

Problem

After use Benchmarksql build 1,000 warehouses ,start runBenchmark. error occurs

11:00:45,448 [Thread-7] ERROR  jTPCCTData : Unexpected SQLException in NEW_ORDER
11:00:45,448 [Thread-7] ERROR  jTPCCTData : [144044c5e7001000-8][10.16.9.12:10069][tpcc1]ERR-CODE: [PXC-10001][ERR_X_PROTOCOL_CLIENT] XDataSource to [email protected]:44881 ERR-CODE: [PXC-10003][ERR_X_PROTOCOL_CONNECTION] XClientPool to [email protected]:44881 get XConnection timeout. 5000000000ns
java.sql.SQLException: [144044c5e7001000-8][10.16.9.12:10069][tpcc1]ERR-CODE: [PXC-10001][ERR_X_PROTOCOL_CLIENT] XDataSource to [email protected]:44881 ERR-CODE: [PXC-10003][ERR_X_PROTOCOL_CONNECTION] XClientPool to [email protected]:44881 get XConnection timeout. 5000000000ns
	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:965)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3973)
	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3909)
	at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2527)
	at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2680)
	at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2484)
	at com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:1858)
	at com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:1966)
	at jTPCCTData.executeNewOrder(jTPCCTData.java:430)
	at jTPCCTData.execute(jTPCCTData.java:95)
	at jTPCCTerminal.executeTransactions(jTPCCTerminal.java:261)
	at jTPCCTerminal.run(jTPCCTerminal.java:88)
	at java.lang.Thread.run(Thread.java:748)

What about the hardware resources allocated for PolarDB-X? This error (XClientPool to [email protected]:44881 get XConnection timeout) might be caused by a DN crash or hang in the case of high concurrency pressure. You can also check the log file for more information.

hardware resources 32c 64G

version: v1
type: polardbx
cluster:
  name: polardb_4dn
  gms:
    image: polardbx/galaxyengine:latest
    host_group: [10.16.9.224]
  cn:
    image: polardbx/galaxysql:latest
    replica: 1
    nodes:
      - host: 10.16.9.12
    resources:
      mem_limit: 20G
  dn:
    image: polardbx/galaxyengine:latest
    replica: 4
    nodes:
      - host_group: [10.16.9.62]
      - host_group: [10.16.9.192]
      - host_group: [10.16.9.120]
      - host_group: [10.16.9.16]
    resources:
      mem_limit: 20G
  cdc:
    image: polardbx/galaxycdc:latest
    replica: 1
    nodes:
      - host: 10.16.9.16
    resources:
      mem_limit: 20G

When running polardb-x cluster ,No other application is running, that is, the server resources are fully allocated to polardb-x cluster.

might be caused by a DN crash or hang in the case of high concurrency pressure. With the above resources, I don't think PolarDB-X can't pressure more than 200 concurrent. Standalone MySQL is fine with the same configuration

wsm12138 avatar May 12 '22 02:05 wsm12138