venus icon indicating copy to clipboard operation
venus copied to clipboard

[venus-auth] Invalid connection and unexpected EOF issues occur / 出现 invalid connection 及 unexpected EOF

Open simlecode opened this issue 2 years ago • 1 comments

链服务模块 / Chain Service Components

  • [ ] venus
  • [X] venus-auth
  • [ ] venus-gateway
  • [ ] venus-messager
  • [ ] venus-miner
  • [ ] 文档 / docs

订单服务模块 / Deal Service Components

  • [ ] venus-market
  • [ ] 文档 / docs

算力服务模块 / Storage Power Service Components

  • [ ] venus-sector-manager
  • [ ] venus-worker
  • [ ] 文档 / docs

描述 / Description

遇到 invalid connection 及 unexpected EOF

[0m[31;1m[260.821ms] [33m[rows:1][35m SELECT count(*) FROM `signers` WHERE `signer` = '3tfser6vgjw5fhb3i4tmjpauiqurcykhmqy2myzbc67lar72tluf7qkjctlon6ys7m5w4kc4znzcqooi6cska' AND `user` = 'cq-21' AND deleted_at IS NULL[0m
[mysql] 2023/05/19 19:31:49 packets.go:37: read tcp 10.123.1.235:41534->47.103.182.80:3306: i/o timeout
[mysql] 2023/05/19 19:31:49 packets.go:37: read tcp 10.123.1.235:41732->47.103.182.80:3306: i/o timeout

2023/05/19 19:31:49 [31;1m/home/runner/work/venus-auth/venus-auth/storage/mysql.go:434 [35;1minvalid connection; invalid connection
[0m[33m[120009.404ms] [34;1m[rows:-][0m SELECT users.* FROM `signers` inner join users on signers.`signer` = '3qyd2itgmzujn4sqsm7lc2muvvnq7yzotxlc7jpqlonjwetnupjii5bxx6jr2ijodyqvdtgczglewskb4mnna' and users.`name` = signers.`user` and users.`is_deleted` = 0 WHERE `signers`.`deleted_at` IS NULL

2023/05/19 19:31:49 [31;1m/home/runner/work/venus-auth/venus-auth/storage/mysql.go:434 [35;1minvalid connection; invalid connection
[0m[33m[120009.638ms] [34;1m[rows:-][0m SELECT users.* FROM `signers` inner join users on signers.`signer` = '3q3gcskttwevlkzxdu4zox7in7cn7wl7x7bbr7bboqtz7ubouxrymh3u5mf6meg666s5revpsn5frnit3ic3q' and users.`name` = signers.`user` and users.`is_deleted` = 0 WHERE `signers`.`deleted_at` IS NULL
[mysql] 2023/05/19 19:31:49 packets.go:37: unexpected EOF

2023/05/19 19:31:49 [31;1m/home/runner/work/venus-auth/venus-auth/storage/mysql.go:434 [35;1minvalid connection; invalid connection
[0m[33m[120009.651ms] [34;1m[rows:-][0m SELECT users.* FROM `signers` inner join users on signers.`signer` = '3rmk4y65ju7rngezeezotup2i7ouu6mssjlmkrhbm64tmaqog6m6ozkes3vrmu56vytuniy4r2hvud6x34tra' and users.`name` = signers.`user` and users.`is_deleted` = 0 WHERE `signers`.`deleted_at` IS NULL
[mysql] 2023/05/19 19:31:49 packets.go:37: unexpected EOF
[mysql] 2023/05/19 19:31:49 packets.go:37: unexpected EOF
[mysql] 2023/05/19 19:31:49 packets.go:37: unexpected EOF
[mysql] 2023/05/19 19:31:49 packets.go:37: unexpected EOF

2023/05/19 19:31:49 [31;1m/home/runner/work/venus-auth/venus-auth/storage/mysql.go:434 [35;1minvalid connection; invalid connection
[0m[33m[120007.053ms] [34;1m[rows:-][0m SELECT users.* FROM `signers` inner join users on signers.`signer` = '3r6m4tkjavy7a5gtyudah3oes3sratlq6g4gwgdee2jtrirsibcayhnd55czr5ig6l7jg3z36cwh6pq5qngaa' and users.`name` = signers.`user` and users.`is_deleted` = 0 WHERE `signers`.`deleted_at` IS NULL
[mysql] 2023/05/19 19:31:49 packets.go:37: unexpected EOF

2023/05/19 19:31:49 [31;1m/home/runner/work/venus-auth/venus-auth/storage/mysql.go:434 [35;1minvalid connection; invalid connection
[0m[33m[120006.872ms] [34;1m[rows:-][0m SELECT users.* FROM `signers` inner join users on signers.`signer` = '3shss47rcdmkm76fgl5r6vlthwupryffj2fkfmv2po6fqyxogoqmv3ua263w2najchlvusvlddp3hsc5mg52a' and users.`name` = signers.`user` and users.`is_deleted` = 0 WHERE `signers`.`deleted_at` IS NULL
[mysql] 2023/05/19 19:31:49 packets.go:37: unexpected EOF

2023/05/19 19:31:49 [31;1m/home/runner/work/venus-auth/venus-auth/storage/mysql.go:434 [35;1minvalid connection; invalid connection
[0m[33m[120007.393ms] [34;1m[rows:-][0m SELECT users.* FROM `signers` inner join users on signers.`signer` = '3r4vi7zl2oxk4gk34x3lnq7kpr6mlxnpgbiorclplhvk5xill5tkojbpnwitc7opwvmukryi3nmoh76fekrka' and users.`name` = signers.`user` and users.`is_deleted` = 0 WHERE `signers`.`deleted_at` IS NULL

2023/05/19 19:31:49 [31;1m/home/runner/work/venus-auth/venus-auth/storage/mysql.go:434 [35;1minvalid connection; invalid connection
[0m[33m[120007.655ms] [34;1m[rows:-][0m SELECT users.* FROM `signers` inner join users on signers.`signer` = '3sqrjmmwyffqw4x6ao5r4ynbwsatj6x3hy22ilvi625piw6cx7ovhwacpedathodcy6fz6jqpkwztqcoj47la' and users.`name` = signers.`user` and users.`is_deleted` = 0 WHERE `signers`.`deleted_at` IS NULL
[mysql] 2023/05/19 19:31:49 packets.go:37: unexpected EOF
[mysql] 2023/05/19 19:31:49 packets.go:37: unexpected EOF

2023/05/19 19:31:49 [31;1m/home/runner/work/venus-auth/venus-auth/storage/mysql.go:434 [35;1minvalid connection; invalid connection
[0m[33m[120007.709ms] [34;1m[rows:-][0m SELECT users.* FROM `signers` inner join users on signers.`signer` = '3qrre5lkrwtstguhjytqmiphhtnhnioxuujnilrvrli2hzi4guk2crcsejdmdutdk3mhgamyc2u62vrcm742q' and users.`name` = signers.`user` and users.`is_deleted` = 0 WHERE `signers`.`deleted_at` IS NULL

详细日志

simlecode avatar May 23 '23 02:05 simlecode

[mysql] 2023/05/19 19:33:49 packets.go:37: unexpected EOF

很明显的网络问题, 读流失败

	var res Result
	var err error
	var isBadConn bool
	for i := 0; i < maxBadConnRetries; i++ {
		res, err = db.exec(ctx, query, args, cachedOrNewConn)
		isBadConn = errors.Is(err, driver.ErrBadConn)
		if !isBadConn {
			break
		}
	}
	if isBadConn {
		return db.exec(ctx, query, args, alwaysNewConn)
	}
	return res, err

驱动层存在重试逻辑, 需要排查网络问题

simlecode avatar May 23 '23 07:05 simlecode