matrixone icon indicating copy to clipboard operation
matrixone copied to clipboard

[Bug]: [date 5.19]tke regression: tpcc 1000w test reported internal error createTxnOpUnsafe due to slow s3 operation

Open heni02 opened this issue 8 months ago • 3 comments

Is there an existing issue for the same bug?

  • [x] I have checked the existing issues.

Branch Name

main

Commit ID

81ee75d7079fc63877d390c175fb99a50c5f03ad

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

job:https://github.com/matrixorigin/mo-nightly-regression/actions/runs/15110800646/job/42480224330

Image mo log: https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22R-n%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-main-nightly-81ee75d70-20250519%5C%22%7D%20%7C%3D%20%60createTxnOpUnsafe%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221747669161224%22,%22to%22:%221747670937224%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

tke tpcc 1000w 1000t

Additional information

No response

heni02 avatar May 20 '25 03:05 heni02

The main issue is that pulling data from S3 is relatively slow. We will continue to monitor to see if this is an isolated incident

XuPeng-SH avatar May 20 '25 05:05 XuPeng-SH

2.2-dev最近两天回归都有报错 5.20号日志: https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22AgG%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-branch-nightly-d6978d00b-20250520%5C%22%7D%20%7C%3D%20%60createTxnOpUnsafe%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221747770700965%22,%22to%22:%221747771495569%22%7D%7D%7D&schemaVersion=1&orgId=1 5.21号日志: https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22AgG%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-branch-nightly-62801a1d5-20250521%5C%22%7D%20%7C%3D%20%60createTxnOpUnsafe%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221747856382520%22,%22to%22:%221747858437387%22%7D%7D%7D&schemaVersion=1&orgId=1

heni02 avatar May 22 '25 02:05 heni02

to 202506

XuPeng-SH avatar Jun 13 '25 02:06 XuPeng-SH

repro: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/16250214372/job/45899633844

log:https://grafana.ci.matrixorigin.cn/explore?panes=%7B%220Y6%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-branch-nightly-621cc6237-20250714%5C%22%7D%20%7C%3D%20%60context%20deadline%20exceeded%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221752479065000%22,%22to%22:%221752482665000%22%7D%7D%7D&schemaVersion=1&orgId=1

Ariznawlll avatar Jul 14 '25 11:07 Ariznawlll