matrixone icon indicating copy to clipboard operation
matrixone copied to clipboard

[Bug]: [2.1.1-hotfix] create ivf index panic:rpc timeout

Open heni02 opened this issue 7 months ago • 4 comments

Is there an existing issue for the same bug?

  • [x] I have checked the existing issues.

Branch Name

2.1.1-hotfix

Commit ID

0c55c1ed6bd090be3138ca18482be7df8825c7f4

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

job:https://github.com/matrixorigin/mo-nightly-regression/actions/runs/15415843392/job/43378576984

Image

mo服务挂了,日志: https://shanghai.idc.matrixorigin.cn:30001/explore?panes=%7B%22mdk%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bhost%3D%5C%2210-222-1-128%5C%22,%20filename%3D%5C%22%2Fdata1%2Frunners%2Faction-runner%2F_work%2Fmo-nightly-regression%2Fmo-nightly-regression%2Fhead%2Fmo-service-0c55c1e-20250603-191744.log%5C%22%7D%20%7C%3D%20%60%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%22now-24h%22,%22to%22:%22now%22%7D%7D%7D&schemaVersion=1&orgId=1

Expected Behavior

No response

Steps to Reproduce

standalone regression gist960 vector benchmark

Additional information

No response

heni02 avatar Jun 04 '25 02:06 heni02

The timeout is caused by excessive disk concurrent IO pressure and long logservice delay.

LeftHandCold avatar Jun 05 '25 02:06 LeftHandCold

单独测试gist960 benchmark没有出现rpc timeout问题了,但create ivf index较2.2性能慢很多,2.1.1-hotfix 创建索引1个小时,2.2只需要10分钟左右 https://github.com/matrixorigin/mo-nightly-regression/actions/runs/15437098132 Image

Image

heni02 avatar Jun 05 '25 03:06 heni02

先降级s0

heni02 avatar Jun 05 '25 03:06 heni02

试了5月份的commit,create index还是很慢,应该之前就有这个问题 Image

heni02 avatar Jun 05 '25 03:06 heni02

2.2-dev和main都没有性能问题了,closed Image

Image

heni02 avatar Aug 01 '25 02:08 heni02