heni02
heni02
工具问题未解决，转到下一个不版本
The above scenario is tested again, and this time the configuration is as follows： set hnsw_threads_build=4 and cn requests/limits values but still appeared CN pod was killed https://shanghai.idc.matrixorigin.cn:30001/d/85a562078cdf77779eaa1add43ccec1e/kubernetes-compute-resources-namespace-pods?orgId=1&var-datasource=prometheus&var-cluster=&var-namespace=mo-hnsw-test&from=1741679496324&to=1741680732357
commit:6c625cc16 cpu 8core,mem 8G,create index cn oom sql: CREATE INDEX hnsw USING hnsw on ann.items_sift (embedding) M = 8 EF_CONSTRUCTION = 200 EF_SEARCH = 64 OP_TYPE 'vector_l2_ops'  https://shanghai.idc.matrixorigin.cn:30001/d/cluster-detail-namespaced/cluster-detail-namespaced?orgId=1&from=1742369643658&to=1742370810740 profile:...
目前资源长期在跑稳定性和故障注入测试，挪到一下版本验证
main commit:81d0a0f9a 验证cn还是被killed，现象也是cn cpu限制8core，实际使用超出8core，内存使用3.25G 配置：
3.0-dev 发布版本：6bb57b13d 配置分别1个cn和2个cn （8core/8G）create hnsw index时都会oom，和之前的现象一致 https://shanghai.idc.matrixorigin.cn:30001/d/cluster-detail-namespaced/cluster-detail-namespaced?orgId=1&var-namespace=mo-hnsw-test&var-account=All&var-interval=$__auto_interval_interval&var-cluster=.%2A&var-loki=loki&from=1756692922345&to=1756700122345 https://shanghai.idc.matrixorigin.cn:30001/explore?panes=%7B%229qa%22:%7B%22datasource%22:%22pyroscope%22,%22queries%22:%5B%7B%22groupBy%22:%5B%5D,%22labelSelector%22:%22%7Bnamespace%3D%5C%22mo-hnsw-test%5C%22,pod%3D%5C%22stability-regression-dis-tp-cn-jwtjg%5C%22%7D%22,%22queryType%22:%22both%22,%22refId%22:%22A%22,%22datasource%22:%7B%22type%22:%22grafana-pyroscope-datasource%22,%22uid%22:%22pyroscope%22%7D,%22profileTypeId%22:%22process_cpu:cpu:nanoseconds:cpu:nanoseconds%22%7D%5D,%22range%22:%7B%22from%22:%221756696345000%22,%22to%22:%221756696717000%22%7D%7D%7D&schemaVersion=1&orgId=1
query时也会oom，cc @iamlinjunhong 是否是一个原因 https://github.com/matrixorigin/matrixone/issues/22465
回归验证resume任务还是会报错 commit: 04f5ee8 https://shanghai.idc.matrixorigin.cn:30001/explore?panes=%7B%22nJ8%22:%7B%22datasource%22:%22loki%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22expr%22:%22%7Bnamespace%3D%5C%22mo-cdc-test%5C%22%7D%20%7C%3D%20%60ERROR%60%22,%22queryType%22:%22range%22,%22datasource%22:%7B%22type%22:%22loki%22,%22uid%22:%22loki%22%7D,%22editorMode%22:%22builder%22%7D%5D,%22range%22:%7B%22from%22:%221737009675134%22,%22to%22:%221737013275134%22%7D%7D%7D&schemaVersion=1&orgId=1 cdc命令： ./mo_cdc task create --task-name "cdc_db" --source-uri="mysql://ac1:admin:[email protected]:6001" --sink-type="mysql" --sink-uri="mysql://root:[email protected]:3306" --databases="tpcc_10:tpcc_10_mysql" --start-ts='2025-01-16T06:50:44+00:00' --end-ts='2025-01-16T10:30:00+00:00' --level="database" [root@mo-srv-128 mo-backup]# ./mo_cdc task pause --task-name "cdc_db" --source-uri="mysql://ac1:admin:[email protected]:6001" OK [root@mo-srv-128 mo-backup]# ./mo_cdc task...