matrixone icon indicating copy to clipboard operation
matrixone copied to clipboard

[Bug]: lots of use of closed network connection error when kill cns during tpch long running test

Open aressu1985 opened this issue 1 year ago • 4 comments

Is there an existing issue for the same bug?

  • [X] I have checked the existing issues.

Branch Name

1.1-dev

Commit ID

7a96a59

Other Environment Information

- Hardware parameters:
3*CN: 16C  64G
1*DN: 16C  64G
3*LOG: 4C 16G
- OS type:
- Others:

Actual Behavior

when running tpch long running queries , if killing one of cns , there lots of error log like: {"level":"ERROR","time":"2024/01/25 11:40:00.991588 +0000","name":"cn-service","caller":"[email protected]/session.go:515","msg":"close conneciton failed","name":"pipeline-server","session-id":518361,"error":"close tcp4 10.10.4.89:6002->10.10.4.88:44858: use of closed network connection"} {"level":"ERROR","time":"2024/01/25 11:40:00.991654 +0000","name":"cn-service","caller":"[email protected]/session.go:515","msg":"close conneciton failed","name":"pipeline-server","session-id":518220,"error":"close tcp4 10.10.4.89:6002->10.10.4.87:41736: use of closed network connection"} {"level":"ERROR","time":"2024/01/25 11:40:00.991882 +0000","name":"cn-service","caller":"[email protected]/session.go:515","msg":"close conneciton failed","name":"pipeline-server","session-id":518302,"error":"close tcp4 10.10.4.89:6002->10.10.4.88:44790: use of closed network connection"} {"level":"ERROR","time":"2024/01/25 11:40:01.074570 +0000","name":"cn-service","caller":"[email protected]/session.go:515","msg":"close conneciton failed","name":"pipeline-server","session-id":518794,"error":"close tcp4 10.10.4.89:6002->10.10.4.87:35240: use of closed network connection"} {"level":"ERROR","time":"2024/01/25 11:40:01.074828 +0000","name":"cn-service","caller":"[email protected]/session.go:515","msg":"close conneciton failed","name":"pipeline-server","session-id":518784,"error":"close tcp4 10.10.4.89:6002->10.10.4.87:35152: use of closed network connection"} {"level":"ERROR","time":"2024/01/25 11:40:01.074919 +0000","name":"cn-service","caller":"[email protected]/session.go:515","msg":"close conneciton failed","name":"pipeline-server","session-id":518019,"error":"close tcp4 10.10.4.89:6002->10.10.4.87:40962: use of closed network connection"} {"level":"ERROR","time":"2024/01/25 11:40:01.074981 +0000","name":"cn-service","caller":"[email protected]/session.go:515","msg":"close conneciton failed","name":"pipeline-server","session-id":518247,"error":"close tcp4 10.10.4.89:6002->10.10.4.87:41992: use of closed network connection"} {"level":"ERROR","time":"2024/01/25 11:40:01.075049 +0000","name":"cn-service","caller":"[email protected]/session.go:515","msg":"close conneciton failed","name":"pipeline-server","session-id":518043,"error":"close tcp4 10.10.4.89:6002->10.10.4.88:43784: use of closed network connection"}

and when cn was restart successfully, this error logs last continously.

closed_network_connection.log

Expected Behavior

No response

Steps to Reproduce

1.run a mo cluster
2.run tpch test
3.kill one of cn

Additional information

No response

aressu1985 avatar Jan 25 '24 11:01 aressu1985

手动kill cn的话 报这个错难道不是正常的吗

badboynt1 avatar Jan 26 '24 10:01 badboynt1

手动kill cn的话 报这个错难道不是正常的吗

问题是,CN重新启动完之后,这个一直在刷个没完没了呀

aressu1985 avatar Jan 27 '24 10:01 aressu1985

无进展,不影响业务的话,可能是某些内部查询等。

m-schen avatar Feb 01 '24 11:02 m-schen

近期无法开展

猜测近期main分支和1.2分支的rpc修改和remote run对它有一定帮助,但是没进行验证。

m-schen avatar Jul 03 '24 10:07 m-schen

近期没有投入这个

m-schen avatar Jul 08 '24 11:07 m-schen

同上一个回复

m-schen avatar Jul 11 '24 11:07 m-schen

This issue has been inactive. Closing to keep the tracker clean. Reopen if still relevant. Thanks!

sukki37 avatar Jul 16 '24 07:07 sukki37