matrixorigin
[Bug]: mo panic from tae when run mixed cases(bvt,tpch,tp)
Is there an existing issue for the same bug?
- [X] I have checked the existing issues.
Environment
- Version or commit-id (e.g. v0.1.0 or 8b23a93):0.5.0
- Hardware parameters:
- OS type:
- Others:
Actual Behavior
{"level":"ERROR","time":"2022/07/10 12:07:16.103357 +0800","caller":"logutil/api.go:63","msg":"clean txn. Get error:tae txn: r-w conflict error txnError:tae txn: r-w conflict error","stacktrace":"github.com/matrixorigin/matrixone/pkg/logutil.Errorf\n\t/home/sudong/matrixone/pkg/logutil/api.go:63\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*TxnHandler).CleanTxn\n\t/home/sudong/matrixone/pkg/frontend/session.go:592\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*MysqlCmdExecutor).doComQuery.func1\n\t/home/sudong/matrixone/pkg/frontend/mysql_cmd_executor.go:1467\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*MysqlCmdExecutor).doComQuery\n\t/home/sudong/matrixone/pkg/frontend/mysql_cmd_executor.go:1773\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*MysqlCmdExecutor).ExecRequest\n\t/home/sudong/matrixone/pkg/frontend/mysql_cmd_executor.go:1900\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*Routine).Loop\n\t/home/sudong/matrixone/pkg/frontend/routine.go:106"} {"level":"ERROR","time":"2022/07/10 12:07:16.103408 +0800","caller":"logutil/api.go:63","msg":"clean txn. Get error:tae txn: r-w conflict error txnError:tae txn: r-w conflict error","stacktrace":"github.com/matrixorigin/matrixone/pkg/logutil.Errorf\n\t/home/sudong/matrixone/pkg/logutil/api.go:63\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*TxnHandler).CleanTxn\n\t/home/sudong/matrixone/pkg/frontend/session.go:592\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*MysqlCmdExecutor).doComQuery.func1\n\t/home/sudong/matrixone/pkg/frontend/mysql_cmd_executor.go:1467\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*MysqlCmdExecutor).doComQuery\n\t/home/sudong/matrixone/pkg/frontend/mysql_cmd_executor.go:1773\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*MysqlCmdExecutor).ExecRequest\n\t/home/sudong/matrixone/pkg/frontend/mysql_cmd_executor.go:1900\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*Routine).Loop\n\t/home/sudong/matrixone/pkg/frontend/routine.go:106"} {"level":"ERROR","time":"2022/07/10 12:07:16.103442 +0800","caller":"logutil/api.go:63","msg":"clean txn. Get error:tae txn: r-w conflict error txnError:tae txn: r-w conflict error","stacktrace":"github.com/matrixorigin/matrixone/pkg/logutil.Errorf\n\t/home/sudong/matrixone/pkg/logutil/api.go:63\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*TxnHandler).CleanTxn\n\t/home/sudong/matrixone/pkg/frontend/session.go:592\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*MysqlCmdExecutor).doComQuery.func1\n\t/home/sudong/matrixone/pkg/frontend/mysql_cmd_executor.go:1467\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*MysqlCmdExecutor).doComQuery\n\t/home/sudong/matrixone/pkg/frontend/mysql_cmd_executor.go:1773\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*MysqlCmdExecutor).ExecRequest\n\t/home/sudong/matrixone/pkg/frontend/mysql_cmd_executor.go:1900\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*Routine).Loop\n\t/home/sudong/matrixone/pkg/frontend/routine.go:106"} {"level":"ERROR","time":"2022/07/10 12:07:16.103472 +0800","caller":"logutil/api.go:63","msg":"clean txn. Get error:tae txn: r-w conflict error txnError:tae txn: r-w conflict error","stacktrace":"github.com/matrixorigin/matrixone/pkg/logutil.Errorf\n\t/home/sudong/matrixone/pkg/logutil/api.go:63\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*TxnHandler).CleanTxn\n\t/home/sudong/matrixone/pkg/frontend/session.go:592\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*MysqlCmdExecutor).doComQuery.func1\n\t/home/sudong/matrixone/pkg/frontend/mysql_cmd_executor.go:1467\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*MysqlCmdExecutor).doComQuery\n\t/home/sudong/matrixone/pkg/frontend/mysql_cmd_executor.go:1773\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*MysqlCmdExecutor).ExecRequest\n\t/home/sudong/matrixone/pkg/frontend/mysql_cmd_executor.go:1900\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*Routine).Loop\n\t/home/sudong/matrixone/pkg/frontend/routine.go:106"} {"level":"ERROR","time":"2022/07/10 12:07:16.103452 +0800","caller":"logutil/api.go:63","msg":"clean txn. Get error:tae txn: r-w conflict error txnError:tae txn: r-w conflict error","stacktrace":"github.com/matrixorigin/matrixone/pkg/logutil.Errorf\n\t/home/sudong/matrixone/pkg/logutil/api.go:63\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*TxnHandler).CleanTxn\n\t/home/sudong/matrixone/pkg/frontend/session.go:592\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*MysqlCmdExecutor).doComQuery.func1\n\t/home/sudong/matrixone/pkg/frontend/mysql_cmd_executor.go:1467\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*MysqlCmdExecutor).doComQuery\n\t/home/sudong/matrixone/pkg/frontend/mysql_cmd_executor.go:1773\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*MysqlCmdExecutor).ExecRequest\n\t/home/sudong/matrixone/pkg/frontend/mysql_cmd_executor.go:1900\ngithub.com/matrixorigin/matrixone/pkg/frontend.(*Routine).Loop\n\t/home/sudong/matrixone/pkg/frontend/routine.go:106"} 2022/07/10 12:07:19 worker exits from a panic: runtime error: index out of range [35963] with length 35963 2022/07/10 12:07:19 worker exits from panic: goroutine 188482876 [running]: github.com/panjf2000/ants/v2.(*goWorker).run.func1.1() /home/sudong/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:58 +0x10c panic({0x124a5c0, 0xc04cb94330}) /usr/lib/golang/src/runtime/panic.go:838 +0x207 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/stl/containers.(*StdVector[...]).Get(...) /home/sudong/matrixone/pkg/vm/engine/tae/stl/containers/stdvec.go:145 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/containers.(*nullableVecImpl[...]).Get(0xc09a546138, 0xc02b50d810?) /home/sudong/matrixone/pkg/vm/engine/tae/containers/vecimpl.go:43 +0x69 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/containers.(*vector[...]).Get(...) /home/sudong/matrixone/pkg/vm/engine/tae/containers/vector.go:138 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/mergesort.Shuffle({0x15142d8, 0xc0e2461f40}, {0xc0a28e6000, 0x8c7c, 0x8c7c?}) /home/sudong/matrixone/pkg/vm/engine/tae/mergesort/func.go:24 +0xe6 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/mergesort.SortBlockColumns({0xc02ba4f3c0, 0x4, 0x3?}, 0x0) /home/sudong/matrixone/pkg/vm/engine/tae/mergesort/mergesort.go:74 +0x59c github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables/jobs.(*compactBlockTask).PrepareData(0xc06e0702d0, {0xc02b50dadc, 0xc, 0xc}) /home/sudong/matrixone/pkg/vm/engine/tae/tables/jobs/compactblk.go:121 +0x4d6 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tables/jobs.(*compactBlockTask).Execute(0xc06e0702d0) /home/sudong/matrixone/pkg/vm/engine/tae/tables/jobs/compactblk.go:155 +0x478 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tasks/ops.(*Op).OnExec(0xc01b29e000) /home/sudong/matrixone/pkg/vm/engine/tae/tasks/ops/ops.go:90 +0x6f github.com/matrixorigin/matrixone/pkg/vm/engine/tae/db.(*ScheduledTxnTask).Execute(0xc110c7def0) /home/sudong/matrixone/pkg/vm/engine/tae/db/task.go:62 +0xfb github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tasks/ops.(*Op).OnExec(0xc00fa93340) /home/sudong/matrixone/pkg/vm/engine/tae/tasks/ops/ops.go:90 +0x6f github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tasks/worker.(*OpWorker).onOp(0xc000276230, {0x7fbd129b76d8, 0xc110c7def0}) /home/sudong/matrixone/pkg/vm/engine/tae/tasks/worker/worker.go:214 +0x30 github.com/matrixorigin/matrixone/pkg/vm/engine/tae/tasks.(*poolHandler).doHandle.func1.1() /home/sudong/matrixone/pkg/vm/engine/tae/tasks/poolhandler.go:59 +0x32 github.com/panjf2000/ants/v2.(*goWorker).run.func1() /home/sudong/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:70 +0x97 created by github.com/panjf2000/ants/v2.(*goWorker).run /home/sudong/go/pkg/mod/github.com/panjf2000/ants/[email protected]/worker.go:48 +0x65
Expected Behavior
No response
Steps to Reproduce
No response
Additional information
No response
We cannot reproduce the specific panic. We reviewed the code and found some bugs, which may cause a similar issue. We fixed bugs by #4027, but we are not entirely sure if this is the root cause of this issue because we cannot reproduce this problem.
it is a problem of stability test, and I trace this type of issue every daoy , when stability is OK , I will process all of these issues.
now sysbench mixed cases test still cause other panics , the this issue will last being in tracking