[Bug]: [date 5.19]tke regression: tpcc 1000w test reported internal error createTxnOpUnsafe due to slow s3 operation
Is there an existing issue for the same bug?
- [x] I have checked the existing issues.
Branch Name
main
Commit ID
81ee75d7079fc63877d390c175fb99a50c5f03ad
Other Environment Information
- Hardware parameters:
- OS type:
- Others:
Actual Behavior
job:https://github.com/matrixorigin/mo-nightly-regression/actions/runs/15110800646/job/42480224330
Expected Behavior
No response
Steps to Reproduce
tke tpcc 1000w 1000t
Additional information
No response
The main issue is that pulling data from S3 is relatively slow. We will continue to monitor to see if this is an isolated incident
2.2-dev最近两天回归都有报错 5.20号日志: https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22AgG%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-branch-nightly-d6978d00b-20250520%5C%22%7D%20%7C%3D%20%60createTxnOpUnsafe%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221747770700965%22,%22to%22:%221747771495569%22%7D%7D%7D&schemaVersion=1&orgId=1 5.21号日志: https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22AgG%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-branch-nightly-62801a1d5-20250521%5C%22%7D%20%7C%3D%20%60createTxnOpUnsafe%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221747856382520%22,%22to%22:%221747858437387%22%7D%7D%7D&schemaVersion=1&orgId=1
to 202506
repro: https://github.com/matrixorigin/mo-nightly-regression/actions/runs/16250214372/job/45899633844
log:https://grafana.ci.matrixorigin.cn/explore?panes=%7B%220Y6%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-branch-nightly-621cc6237-20250714%5C%22%7D%20%7C%3D%20%60context%20deadline%20exceeded%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221752479065000%22,%22to%22:%221752482665000%22%7D%7D%7D&schemaVersion=1&orgId=1