matrixone icon indicating copy to clipboard operation
matrixone copied to clipboard

[Bug]: [date 3.23]tke regression: sysbench test after reported Duplicate entry error not exit ,always hung

Open heni02 opened this issue 1 year ago • 9 comments

Is there an existing issue for the same bug?

  • [X] I have checked the existing issues.

Branch Name

main

Commit ID

2674995df

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

job:https://github.com/matrixorigin/mo-nightly-regression/actions/runs/8403426531/job/23017476777 企业微信截图_c562055a-fb0d-4393-9172-e52e56d25847

之前也出现过Duplicate entry错误,测试会直接退出不再继续运行,但最近几天报了该错误后无法退出一直hung,怀疑很多session没有closed,需要定位

log: https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22WhO%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-nightly-regression-20240323%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221711238284000%22,%22to%22:%221711238338000%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

sysbench 10w read_write test

Additional information

No response

heni02 avatar Mar 25 '24 08:03 heni02

@ck89119 帮忙看下。

daviszhen avatar Mar 25 '24 08:03 daviszhen

date 5.3 tke regression https://github.com/matrixorigin/mo-nightly-regression/actions/runs/8940629660/job/24569679599 企业微信截图_d082730a-f102-4801-b42f-5a7ab21b7917 mo log: https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22Qq4%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-nightly-regression-20240503%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221714768123000%22,%22to%22:%221714771808000%22%7D%7D%7D&schemaVersion=1&orgId=1

五一期间出现的问题,非必现,profile自动清理掉了,等下次复现在定位

heni02 avatar May 06 '24 06:05 heni02

date 5.16 regression 复现,报了dup后现象mo hung住了 job:https://github.com/matrixorigin/mo-nightly-regression/actions/runs/9114235510/job/25069419376 企业微信截图_6b401982-a913-4c3f-b7d7-ef1581f816e3

mo log: https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22C5l%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-nightly-regression-20240516%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221715890350000%22,%22to%22:%221715892044000%22%7D%7D%7D&schemaVersion=1&orgId=1

profile: etl_profile.zip

heni02 avatar May 17 '24 02:05 heni02

Selection_014 Selection_015

daviszhen avatar May 17 '24 02:05 daviszhen

https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22C5l%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-nightly-regression-20240516%5C%22%7D%20%7C%3D%20%60found%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221715890350000%22,%22to%22:%221715893184000%22%7D%7D%7D&schemaVersion=1&orgId=1

long running的

Selection_016 Selection_017

daviszhen avatar May 17 '24 02:05 daviszhen

leak 今天会把FOOTPRINT加上。

daviszhen avatar May 17 '24 02:05 daviszhen

https://grafana.ci.matrixorigin.cn/goto/z4ewk7PIR?orgId=1

zhangxu19830126 avatar May 17 '24 03:05 zhangxu19830126

@daviszhen 上面这个是卡住在等锁的线程,在等待的事务,所有等待的事务都是leak的那个事务。是leak引起的

zhangxu19830126 avatar May 17 '24 03:05 zhangxu19830126

txn leak:018f8309b98a701784181859b15fb371

20:13:27 sql failed, but txn is still active

@daviszhen please help to resolve this issue

image

https://grafana.ci.matrixorigin.cn/explore?panes=%7B%22C5l%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-nightly-regression-20240516%5C%22%7D%20%7C%3D%20%60018f8309b98a701784181859b15fb371%60%20%21%3D%20%60wait%20too%20long%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221715890350000%22,%22to%22:%221715893184000%22%7D%7D%7D&schemaVersion=1&orgId=1

sukki37 avatar May 17 '24 03:05 sukki37

duplicate的问题,可能跟 #15843 是一个事情 huang的问题不确定

ouyuanning avatar May 20 '24 05:05 ouyuanning

这个需要测试脚本去修改。

daviszhen avatar May 23 '24 10:05 daviszhen

未投入

daviszhen avatar May 28 '24 10:05 daviszhen

未投入

daviszhen avatar May 31 '24 15:05 daviszhen

未投入

daviszhen avatar Jun 05 '24 13:06 daviszhen