Is there an existing issue for the same bug?

[X] I have checked the existing issues.

Branch Name

1.1-dev

Commit ID

b90bf97fe

Other Environment Information

- Hardware parameters:
- OS type:
- Others:

Actual Behavior

Insert less than 200 rows and get the following result:

2024/01/03 12:01:53.735266 +0000 WARN [email protected]/finisher_api.go:647 trace {"elapsed": "2m6.871143051s", "rows": 196, "sql": "INSERT INTO `mo_catalog`.`statement_cu` (`statement_id`,`account`,`response_at`,`cu`,`account_id`) VALUES (...)

Expected Behavior

The query succeeds in short time

Steps to Reproduce

NA

Additional information

Dev env For detailed schema and SQLs, refer to https://github.com/matrixorigin/MO-Cloud/issues/2200

Jan 03 '24 12:01 DanielZhangQD

MySQL [mo_cloud]> select statement_id,status,duration from system.statement_info where statement like '%INSERT INTO `mo_catalog`.`statement_cu`%' and request_at >= '2024-01-03 12:00:00' order by duration desc;
+--------------------------------------+---------+--------------+
| statement_id                         | status  | duration     |
+--------------------------------------+---------+--------------+
| 40a1f570-aa32-11ee-938e-729694d477be | Success | 148216126487 |
| 95c37f18-aa31-11ee-938e-729694d477be | Success | 125535215584 |
| 200af7e8-aa35-11ee-938e-729694d477be | Success | 122606068196 |
| 4aabe79e-aa33-11ee-938e-729694d477be | Success | 117256103508 |

Jan 03 '24 12:01 DanielZhangQD

我在dev环境上复制了一张statement_cu表出来, 然后模拟https://github.com/matrixorigin/MO-Cloud/issues/2200的方式往里面插入数据, 并用explain analyze 查看每个节点的耗时情况, 耗时最长的是table scan, 之前的慢有没有可能是s3的时延?

Jan 04 '24 08:01 jensenojs

利用@jensenojs 的环境，执行 insert:

另起一个 CN，执行 insert，结果如下：

最新的 main 镜像, commit id: 164ddc1969c3aa425e85f401822dd5ea4c927351, 有 log
同集群相同 commit id,，结果如下：
最新的 main, 镜像, commit id: 164ddc1969c3aa425e85f401822dd5ea4c927351, 无 log
最新的 1.1，commit id: 35071af36d952ed5618c975d33137a1f6f8fe2de

所以，很有可能，main 已经修复了这个问题。

Jan 05 '24 16:01 gouhongshen

select into statement_cu_for_test values (...)

after tryFastRanges 指没有命中 try fast ranges

after tryFastRanges] blkHit: 5596, blkTotal: 5596, blkHitRate: 1.000000
slow-insert: [after tryFastRanges] blkHit: 5596, blkTotal: 5596, blkHitRate: 1.000000"}

[after tryFastRanges] blkHit: 1895, blkTotal: 5596, blkHitRate: 0.338635
slow-insert: [after tryFastRanges] blkHit: 1895, blkTotal: 5596, blkHitRate: 0.338635"}

|

insert into statement_cu_for_test values(1111112, "", "", 0, 2222222224);

[in tryFastRanges] blkHit: 11, blkTotal: 5596, blkHitRate: 0.001966
slow-insert: [in tryFastRanges] blkHit: 11, blkTotal: 5596, blkHitRate: 0.001966"}

Jan 05 '24 17:01 gouhongshen

select into statement_cu_for_test values (...)

after tryFastRanges 指没有命中 try fast ranges

after tryFastRanges] blkHit: 5596, blkTotal: 5596, blkHitRate: 1.000000
slow-insert: [after tryFastRanges] blkHit: 5596, blkTotal: 5596, blkHitRate: 1.000000"}

[after tryFastRanges] blkHit: 1895, blkTotal: 5596, blkHitRate: 0.338635
slow-insert: [after tryFastRanges] blkHit: 1895, blkTotal: 5596, blkHitRate: 0.338635"}

|

insert into statement_cu_for_test values(1111112, "", "", 0, 2222222224);

[in tryFastRanges] blkHit: 11, blkTotal: 5596, blkHitRate: 0.001966
slow-insert: [in tryFastRanges] blkHit: 11, blkTotal: 5596, blkHitRate: 0.001966"}

Why didn't it hit the try fast range？

Jan 06 '24 03:01 sukki37

目前有一个简单的结论：

为什么没有走 fast ranges (bloom filter)？ INSERT INTO mo_catalog.statement_cu_for_test (statement_id, account,response_at,cu,account_id) VALUES (...) 的 expression 包含 in 操作， ranges 里面的 fast ranges 还不支持 in 操作。

(a=x1 and b=y1) or (a=x2 and b = y2) or (a=x3 and b=y3) ...

ranges 还有 zone map 过滤，为什么没有发挥太大作用？经过测试，发现 statement_cu 表的 cluster key 分布太均匀，4千万数据，account_id 只有 46 个不同的值，但 statement_id 是uuid，ndv 应该很高。

选择率低这个，有个佐证，随机 select statement_id in (x, y) and account_id = z，会返回 7% 的 block，选择 8 条数据，就会返回 11% 的 block. 这一点应该还需要计算组的同事确认下。

select statement_id,account_id from mo_catalog.statement_cu_for_test where statement_id in ("af0a75c1-9580-11ee-9696-2ada383c6298", "af0bd750-9580-11ee-9696-2ada383c6298") and account_id = 153000001;

|

select statement_id,account_id from mo_catalog.statement_cu_for_test where statement_id in ("caad5be4-9580-11ee-9696-2ada383c6298", "caad6a9f-9580-11ee-9696-2ada383c6298", "caad784b-9580-11ee-9696-2ada383c6298", "caad7a28-9580-11ee-9696-2ada383c6298", "b65906ce-97c7-11ee-93c8-7209a4de7f7c", "b6590788-97c7-11ee-93c8-7209a4de7f7c", "b659225e-97c7-11ee-93c8-7209a4de7f7c", "b6592329-97c7-11ee-93c8-7209a4de7f7c") and account_id = 153000001

Jan 06 '24 07:01 gouhongshen

exec_plan.json

table scan on statement_cu costs over 100s. A strange point is that filter condition in this step does not contain request_at or response_at

mysql> select request_at, duration, statement_id,status,duration,exec_plan from system.statement_info where request_at >= '2024-01-09 05:19:00' and statement_id='17f07b41-aeaf-11ee-85a4-ba4749d34a35' ; +----------------------------+--------------+--------------------------------------+---------+--------------+--------------------------------------------------------------------------------------------------------------------------------------+ | 2024-01-09 05:22:33.920988 | 104718945555 | 17f07b41-aeaf-11ee-85a4-ba4749d34a35 | Success | 104718945555 | {"steps":[{"graphData":{"nodes":[{"id":"5","name":"Sink","title":"sink","labels":[{"name":"Sink","value":[]}],"statistics":{"Time":[{"name":"Time Consumed","value":3923410,"unit":"ns"},{"name":"Wait Time","value":207574597282,"unit":"ns"},{"name":"Scan Time","value":0,"unit":"ns"},{"name":"Insert Time","value":1376941,"unit":"ns"}],"Memory":[{"name":"Memory Size","value":103295,"unit":"byte"}],"Throughput":[{"name":"Input Rows","value":566,"unit":"count"},{"name":"Output Rows","value":0,"unit":"count"},{"name":"Input Size","value":103295,"unit":"byte"},{"name":"Output Size","value":0,"unit":"byte"}],"IO":[{"name":"Disk IO","value":0,"unit":"byte"},{"name":"S3 IO Byte","value":0,"unit":"byte"},{"name":"S3 IO Input Count","value":0,"unit":"count"},{"name":"S3 IO Output Count","value":0,"unit":"count"}],"Network":[{"name":"Network","value":0,"unit":"byte"}]},"stats":{"blocknum":1,"outcnt":1000,"cost":1000,"hashmapsize":0,"rowsize":0},"totalStats":{"name":"Time spent","value":3923410,"unit":"ns"}},{"id":"4","name":"Lock Operator","title":"lock_op","labels":[{"name":"Lock op","value":[]}],"statistics":{"Time":[{"name":"Time Consumed","value":0,"unit":"ns"},{"name":"Wait Time","value":0,"unit":"ns"},{"name":"Scan Time","value":0,"unit":"ns"},{"name":"Insert Time","value":0,"unit":"ns"}],"Memory":[{"name":"Memory Size","value":0,"unit":"byte"}],"Throughput":[{"name":"Input Rows","value":0,"unit":"count"},{"name":"Output Rows","value":0,"unit":"count"},{"name":"Input Size","value":0,"unit":"byte"},{"name":"Output Size","value":0,"unit":"byte"}],"IO":[{"name":"Disk IO","value":0,"unit":"byte"},{"name":"S3 IO Byte","value":0,"unit":"byte"},{"name":"S3 IO Input Count","value":0,"unit":"count"},{"name":"S3 IO Output Count","value":0,"unit":"count"}],"Network":[{"name":"Network","value":0,"unit":"byte"}]},"stats":{"blocknum":1,"outcnt":1000,"cost":1000,"hashmapsize":0,"rowsize":0},"totalStats":{"name":"Time spent","value":0,"unit":"ns"}},{"id":"3","name":"Pre Insert","title":"preinsert","labels":[{"name":"Pre insert","value":[]}],"statistics":{"Time":[{"name":"Time Consumed","value":0,"unit":"ns"},{"name":"Wait Time","value":0,"unit":"ns"},{"name":"Scan Time","value":0,"unit":"ns"},{"name":"Insert Time","value":0,"unit":"ns"}],"Memory":[{"name":"Memory Size","value":0,"unit":"byte"}],"Throughput":[{"name":"Input Rows","value":0,"unit":"count"},{"name":"Output Rows","value":0,"unit":"count"},{"name":"Input Size","value":0,"unit":"byte"},{"name":"Output Size","value":0,"unit":"byte"}],"IO":[{"name":"Disk IO","value":0,"unit":"byte"},{"name":"S3 IO Byte","value":0,"unit":"byte"},{"name":"S3 IO Input Count","value":0,"unit":"count"},{"name":"S3 IO Output Count","value":0,"unit":"count"}],"Network":[{"name":"Network","value":0,"unit":"byte"}]},"stats":{"blocknum":1,"outcnt":1000,"cost":1000,"hashmapsize":0,"rowsize":0},"totalStats":{"name":"Time spent","value":0,"unit":"ns"}},{"id":"2","name":"Project","title":"#[0,0], #[0,1], #[0,2], #[0,3], #[0,4]","labels":[{"name":"List of expressions","value":["#[0,0]","#[0,1]","#[0,2]","#[0,3]","#[0,4]"]}],"statistics":{"Time":[{"name":"Time Consumed","value":0,"unit":"ns"},{"name":"Wait Time","value":0,"unit":"ns"},{"name":"Scan Time","value":0,"unit":"ns"},{"name":"Insert Time","value":0,"unit":"ns"}],"Memory":[{"name":"Memory Size","value":0,"unit":"byte"}],"Throughput":[{"name":"Input Rows","value":0,"unit":"count"},{"name":"Output Rows","value":0,"unit":"count"},{"name":"Input Size","value":0,"unit":"byte"},{"name":"Output Size","value":0,"unit":"byte"}],"IO":[{"name":"Disk IO","value":0,"unit":"byte"},{"name":"S3 IO Byte","value":0,"unit":"byte"},{"name":"S3 IO Input Count","value":0,"unit":"count"},{"name":"S3 IO Output Count","value":0,"unit":"count"}],"Network":[{"name":"Network","value":0,"unit":"byte"}]},"stats":{"blocknum":1,"outcnt":1000,"cost":1000,"hashmapsize":0,"rowsize":0},"totalStats":{"name":"Time spent","value":0,"unit":"ns"}},{"id":"1","name":"Project","title":"_ValueScan.column_0, _ValueScan.column_1, _ValueScan.column_2, _ValueScan.column_3, _ValueScan.column_4","labels":[{"name":"List of expressions","value":["_ValueScan.column_0","_ValueScan.column_1","_ValueScan.column_2","_ValueScan.column_3","_ValueScan.column_4"]}],"statistics":{"Time":[{"name":"Time Consumed","value":2724,"unit":"ns"},{"name":"Wait Time","value":0,"unit":"ns"},{"name":"Scan Time","value":0,"unit":"ns"},{"name":"Insert Time","value":0,"unit":"ns"}],"Memory":[{"name":"Memory Size","value":41884,"unit":"byte"}],"Throughput":[{"name":"Input Rows","value":283,"unit":"count"},{"name":"Output Rows","value":283,"unit":"count"},{"name":"Input Size","value":41884,"unit":"byte"},{"name":"Output Size","value":41884,"unit":"byte"}],"IO":[{"name":"Disk IO","value":0,"unit":"byte"},{"name":"S3 IO Byte","value":0,"unit":"byte"},{"name":"S3 IO Input Count","value":0,"unit":"count"},{"name":"S3 IO Output Count","value":0,"unit":"count"}],"Network":[{"name":"Network","value":0,"unit":"byte"}]},"stats":{"blocknum":1,"outcnt":1000,"cost":1000,"hashmapsize":0,"rowsize":0},"totalStats":{"name":"Time spent","value":2724,"unit":"ns"}},{"id":"0","name":"Values Scan","title":"column_0, column_1, column_2, column_3, column_4","labels":[{"name":"List of values","value":["column_0","column_1","column_2","column_3","column_4"]}],"statistics":{"Time":[{"name":"Time Consumed","value":462015,"unit":"ns"},{"name":"Wait Time","value":2078982,"unit":"ns"},{"name":"Scan Time","value":0,"unit":"ns"},{"name":"Insert Time","value":0,"unit":"ns"}],"Memory":[{"name":"Memory Size","value":41884,"unit":"byte"}],"Throughput":[{"name":"Input Rows","value":283,"unit":"count"},{"name":"Output Rows","value":283,"unit":"count"},{"name":"Input Size","value":21508,"unit":"byte"},{"name":"Output Size","value":41884,"unit":"byte"}],"IO":[{"name":"Disk IO","value":0,"unit":"byte"},{"name":"S3 IO Byte","value":0,"unit":"byte"},{"name":"S3 IO Input Count","value":0,"unit":"count"},{"name":"S3 IO Output Count","value":0,"unit":"count"}],"Network":[{"name":"Network","value":0,"unit":"byte"}]},"stats":{"blocknum":1,"outcnt":1000,"cost":1000,"hashmapsize":0,"rowsize":0},"totalStats":{"name":"Time spent","value":462015,"unit":"ns"}}],"edges":[{"id":"E4","src":"4","dst":"5","output":0,"unit":"count"},{"id":"E3","src":"3","dst":"4","output":0,"unit":"count"},{"id":"E2","src":"2","dst":"3","output":0,"unit":"count"},{"id":"E1","src":"1","dst":"2","output":283,"unit":"count"},{"id":"E0","src":"0","dst":"1","output":283,"unit":"count"}],"labels":[],"global":{"statistics":{"Time":[{"name":"Time Consumed","value":4388149,"unit":"ns"},{"name":"Wait Time","value":207576676264,"unit":"ns"}],"Memory":[{"name":"Memory Size","value":187063,"unit":"byte"}],"Throughput":[{"name":"Input Rows","value":1132,"unit":"count"},{"name":"Output Rows","value":566,"unit":"count"},{"name":"Input Size","value":166687,"unit":"byte"},{"name":"Output Size","value":83768,"unit":"byte"}],"IO":[{"name":"Disk IO","value":0,"unit":"byte"},{"name":"S3 IO Byte","value":0,"unit":"byte"},{"name":"S3 IO Input Count","value":0,"unit":"count"},{"name":"S3 IO Output Count","value":0,"unit":"count"}],"Network":[{"name":"Network","value":0,"unit":"byte"}]},"totalStats":{"name":"Time spent","value":4388149,"unit":"ns"}}},"step":0,"description":"","state":"success","stats":{}},{"graphData":{"nodes":[{"id":"7","name":"Insert","title":"mo_catalog.statement_cu","labels":[{"name":"Full table name","value":"mo_catalog.statement_cu"}],"statistics":{"Time":[{"name":"Time Consumed","value":0,"unit":"ns"},{"name":"Wait Time","value":0,"unit":"ns"},{"name":"Scan Time","value":0,"unit":"ns"},{"name":"Insert Time","value":0,"unit":"ns"}],"Memory":[{"name":"Memory Size","value":0,"unit":"byte"}],"Throughput":[{"name":"Input Rows","value":0,"unit":"count"},{"name":"Output Rows","value":0,"unit":"count"},{"name":"Input Size","value":0,"unit":"byte"},{"name":"Output Size","value":0,"unit":"byte"}],"IO":[{"name":"Disk IO","value":0,"unit":"byte"},{"name":"S3 IO Byte","value":0,"unit":"byte"},{"name":"S3 IO Input Count","value":0,"unit":"count"},{"name":"S3 IO Output Count","value":0,"unit":"count"}],"Network":[{"name":"Network","value":0,"unit":"byte"}]},"stats":{"blocknum":1,"outcnt":1000,"cost":1000,"hashmapsize":0,"rowsize":0},"totalStats":{"name":"Time spent","value":0,"unit":"ns"}},{"id":"6","name":"Sink Scan","title":"sink_scan","labels":[{"name":"Sink scan","value":[]}],"statistics":{"Time":[{"name":"Time Consumed","value":0,"unit":"ns"},{"name":"Wait Time","value":0,"unit":"ns"},{"name":"Scan Time","value":0,"unit":"ns"},{"name":"Insert Time","value":0,"unit":"ns"}],"Memory":[{"name":"Memory Size","value":0,"unit":"byte"}],"Throughput":[{"name":"Input Rows","value":0,"unit":"count"},{"name":"Output Rows","value":0,"unit":"count"},{"name":"Input Size","value":0,"unit":"byte"},{"name":"Output Size","value":0,"unit":"byte"}],"IO":[{"name":"Disk IO","value":0,"unit":"byte"},{"name":"S3 IO Byte","value":0,"unit":"byte"},{"name":"S3 IO Input Count","value":0,"unit":"count"},{"name":"S3 IO Output Count","value":0,"unit":"count"}],"Network":[{"name":"Network","value":0,"unit":"byte"}]},"stats":{"blocknum":1,"outcnt":1000,"cost":1000,"hashmapsize":0,"rowsize":0},"totalStats":{"name":"Time spent","value":0,"unit":"ns"}}],"edges":[{"id":"E6","src":"6","dst":"7","output":0,"unit":"count"}],"labels":[],"global":{"statistics":{"Time":[{"name":"Time Consumed","value":0,"unit":"ns"},{"name":"Wait Time","value":0,"unit":"ns"}],"Memory":[{"name":"Memory Size","value":0,"unit":"byte"}],"Throughput":[{"name":"Input Rows","value":0,"unit":"count"},{"name":"Output Rows","value":0,"unit":"count"},{"name":"Input Size","value":0,"unit":"byte"},{"name":"Output Size","value":0,"unit":"byte"}],"IO":[{"name":"Disk IO","value":0,"unit":"byte"},{"name":"S3 IO Byte","value":0,"unit":"byte"},{"name":"S3 IO Input Count","value":0,"unit":"count"},{"name":"S3 IO Output Count","value":0,"unit":"count"}],"Network":[{"name":"Network","value":0,"unit":"byte"}]},"totalStats":{"name":"Time spent","value":0,"unit":"ns"}}},"step":1,"description":"","state":"success","stats":{}},{"graphData":{"nodes":[{"id":"10","name":"Fuzzy filter","title":"fuzzy_filter","labels":[{"name":"Fuzzy filter","value":[]}],"statistics":{"Time":[{"name":"Time Consumed","value":0,"unit":"ns"},{"name":"Wait Time","value":0,"unit":"ns"},{"name":"Scan Time","value":0,"unit":"ns"},{"name":"Insert Time","value":0,"unit":"ns"}],"Memory":[{"name":"Memory Size","value":0,"unit":"byte"}],"Throughput":[{"name":"Input Rows","value":0,"unit":"count"},{"name":"Output Rows","value":0,"unit":"count"},{"name":"Input Size","value":0,"unit":"byte"},{"name":"Output Size","value":0,"unit":"byte"}],"IO":[{"name":"Disk IO","value":0,"unit":"byte"},{"name":"S3 IO Byte","value":0,"unit":"byte"},{"name":"S3 IO Input Count","value":0,"unit":"count"},{"name":"S3 IO Output Count","value":0,"unit":"count"}],"Network":[{"name":"Network","value":0,"unit":"byte"}]},"stats":{"blocknum":1,"outcnt":1000,"cost":1000,"hashmapsize":0,"rowsize":0},"totalStats":{"name":"Time spent","value":0,"unit":"ns"}},{"id":"9","name":"Table Scan","title":"mo_catalog.statement_cu","labels":[{"name":"Full table name","value":"mo_catalog.statement_cu"},{"name":"Columns","value":["statement_id","account_id","__mo_cpkey_col"]},{"name":"Total columns","value":5},{"name":"Scan columns","value":3},{"name":"Filter conditions","value":["((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED)))) or ((account_id = cast(? AS INT UNSIGNED)) and (statement_id = cast(? AS VARCHAR)))) or ((statement_id = cast(? AS VARCHAR)) and (account_id = cast(? AS INT UNSIGNED))))"]}],"statistics":{"Time":[{"name":"Time Consumed","value":824118056991,"unit":"ns"},{"name":"Wait Time","value":103786091863,"unit":"ns"},{"name":"Scan Time","value":0,"unit":"ns"},{"name":"Insert Time","value":0,"unit":"ns"}],"Memory":[{"name":"Memory Size","value":1578259339,"unit":"byte"}],"Throughput":[{"name":"Input Rows","value":11777416,"unit":"count"},{"name":"Output Rows","value":0,"unit":"count"},{"name":"Input Size","value":1566481923,"unit":"byte"},{"name":"Output Size","value":0,"unit":"byte"}],"IO":[{"name":"Disk IO","value":0,"unit":"byte"},{"name":"S3 IO Byte","value":1566481923,"unit":"byte"},{"name":"S3 IO Input Count","value":0,"unit":"count"},{"name":"S3 IO Output Count","value":0,"unit":"count"}],"Network":[{"name":"Network","value":0,"unit":"byte"}]},"stats":{"blocknum":6138,"outcnt":45179232,"cost":45179232,"hashmapsize":0,"rowsize":0},"totalStats":{"name":"Time spent","value":824118056991,"unit":"ns"}},{"id":"8","name":"Sink Scan","title":"sink_scan","labels":[{"name":"Sink scan","value":[]}],"statistics":{"Time":[{"name":"Time Consumed","value":15070,"unit":"ns"},{"name":"Wait Time","value":0,"unit":"ns"},{"name":"Scan Time","value":0,"unit":"ns"},{"name":"Insert Time","value":0,"unit":"ns"}],"Memory":[{"name":"Memory Size","value":19527,"unit":"byte"}],"Throughput":[{"name":"Input Rows","value":283,"unit":"count"},{"name":"Output Rows","value":283,"unit":"count"},{"name":"Input Size","value":61411,"unit":"byte"},{"name":"Output Size","value":19527,"unit":"byte"}],"IO":[{"name":"Disk IO","value":0,"unit":"byte"},{"name":"S3 IO Byte","value":0,"unit":"byte"},{"name":"S3 IO Input Count","value":0,"unit":"count"},{"name":"S3 IO Output Count","value":0,"unit":"count"}],"Network":[{"name":"Network","value":0,"unit":"byte"}]},"stats":{"blocknum":1,"outcnt":1000,"cost":1000,"hashmapsize":0,"rowsize":0},"totalStats":{"name":"Time spent","value":15070,"unit":"ns"}}],"edges":[{"id":"E9","src":"9","dst":"10","output":0,"unit":"count"},{"id":"E8","src":"8","dst":"10","output":283,"unit":"count"}],"labels":[],"global":{"statistics":{"Time":[{"name":"Time Consumed","value":824118072061,"unit":"ns"},{"name":"Wait Time","value":103786091863,"unit":"ns"}],"Memory":[{"name":"Memory Size","value":1578278866,"unit":"byte"}],"Throughput":[{"name":"Input Rows","value":11777699,"unit":"count"},{"name":"Output Rows","value":283,"unit":"count"},{"name":"Input Size","value":1566543334,"unit":"byte"},{"name":"Output Size","value":19527,"unit":"byte"}],"IO":[{"name":"Disk IO","value":0,"unit":"byte"},{"name":"S3 IO Byte","value":1566481923,"unit":"byte"},{"name":"S3 IO Input Count","value":0,"unit":"count"},{"name":"S3 IO Output Count","value":0,"unit":"count"}],"Network":[{"name":"Network","value":0,"unit":"byte"}]},"totalStats":{"name":"Time spent","value":824118072061,"unit":"ns"}}},"step":2,"description":"","state":"success","stats":{}}],"code":0,"message":"","uuid":"17f07b41-aeaf-11ee-85a4-ba4749d34a35","NewPlanStats":{"ParseDuration":16425,"PlanDuration":4913,"CompileDuration":931804892,"ExecutionDuration":103786966303,"ParseStartTime":"2024-01-09T05:22:33.920988931Z","PlanStartTime":"2024-01-09T05:22:33.921085392Z","CompileStartTime":"2024-01-09T05:22:33.921082704Z","ExecutionStartTime":"2024-01-09T05:22:34.852890665Z","ExecutionEndTime":"2024-01-09T05:24:18.639856955Z"}} | +----------------------------+--------------+--------------------------------------+---------+--------------+--------------------------------------------------------------------------------------------------------------------------------------+ 1 row in set (0.09 sec)

Jan 09 '24 07:01 sukki37

@sukki37 那个是多核的cpu时间，不是用户感知的那个时间。如果table scan是并行的话。那个时间应该是是加起来了。比如 10条pipeline并行执行table_scan，每条线执行了2ms ，那么这里就是10*2ms了。

Jan 09 '24 07:01 ouyuanning

经过一些测试，有一些现象： cluster primary key (a bigint, b bigint), 现有 1000W 数据， batch Insert (100 values)，插入 100 W 行。

|

从上面的图可以看出来，事务的时间都耗在了 compile.Run，下面的图是compile.Run 里面算子的耗时：

|

另外补充下，每 100 次 Insert 的平均耗时：

terminal: 0, insert 10000, 100 each time
[0_100):    23.590000ms
[100_200):    30.500000ms
[200_300):    41.120000ms
[300_400):    53.240000ms
[400_500):    64.540000ms
[500_600):    76.140000ms
[600_700):    88.110000ms
[700_800):   100.560000ms
[800_900):   111.980000ms
[900_1000):   124.830000ms
[1000_1100):   136.530000ms
[1100_1200):   148.640000ms
[1200_1300):   160.390000ms
[1300_1400):   172.300000ms
[1400_1500):   184.720000ms
[1500_1600):   197.070000ms
[1600_1700):   209.060000ms
[1700_1800):   221.220000ms
[1800_1900):   233.100000ms
[1900_2000):   245.430000ms
[2000_2100):   257.690000ms
[2100_2200):   269.370000ms
[2200_2300):    29.780000ms
[2300_2400):    30.770000ms
[2400_2500):    41.130000ms
[2500_2600):    52.700000ms
[2600_2700):    64.840000ms
[2700_2800):    76.230000ms
[2800_2900):    88.810000ms
[2900_3000):   100.210000ms
[3000_3100):   112.750000ms
[3100_3200):   124.910000ms
[3200_3300):   136.820000ms
[3300_3400):   148.800000ms
[3400_3500):   161.150000ms
[3500_3600):   173.250000ms
[3600_3700):   185.330000ms
[3700_3800):   197.380000ms
[3800_3900):   209.680000ms
[3900_4000):   221.720000ms
[4000_4100):   234.020000ms
[4100_4200):   245.870000ms
[4200_4300):   258.410000ms
[4300_4400):   250.530000ms
[4400_4500):    26.080000ms
[4500_4600):    33.860000ms
[4600_4700):    42.300000ms
[4700_4800):    54.780000ms
[4800_4900):    65.740000ms
[4900_5000):    77.780000ms
[5000_5100):    90.010000ms
[5100_5200):   102.280000ms
[5200_5300):   113.880000ms
[5300_5400):   125.930000ms
[5400_5500):   138.380000ms
[5500_5600):   150.540000ms
[5600_5700):   162.510000ms
[5700_5800):   174.830000ms
[5800_5900):   186.620000ms
[5900_6000):   198.640000ms
[6000_6100):   210.620000ms
[6100_6200):   223.080000ms
[6200_6300):   235.480000ms
[6300_6400):   247.410000ms
[6400_6500):   137.950000ms
[6500_6600):    27.220000ms
[6600_6700):    36.070000ms
[6700_6800):    47.780000ms
[6800_6900):    59.500000ms
[6900_7000):    71.590000ms
[7000_7100):    83.090000ms
[7100_7200):    94.990000ms
[7200_7300):   107.270000ms
[7300_7400):   119.370000ms
[7400_7500):   131.840000ms
[7500_7600):   144.020000ms
[7600_7700):   155.600000ms
[7700_7800):   167.850000ms
[7800_7900):   179.930000ms
[7900_8000):   191.970000ms
[8000_8100):   204.070000ms
[8100_8200):   216.450000ms
[8200_8300):   228.420000ms
[8300_8400):   240.770000ms
[8400_8500):   252.820000ms
[8500_8600):    40.910000ms
[8600_8700):    32.470000ms
[8700_8800):    41.340000ms
[8800_8900):    53.180000ms
[8900_9000):    63.890000ms
[9000_9100):    76.090000ms
[9100_9200):    88.680000ms
[9200_9300):   100.650000ms
[9300_9400):   112.240000ms
[9400_9500):   124.480000ms
[9500_9600):   136.740000ms
[9600_9700):   148.540000ms
[9700_9800):   160.280000ms
[9800_9900):   173.060000ms
[9900_10000):   185.500000ms

Jan 12 '24 10:01 gouhongshen

1、receiveFromSingleReg 是等待table_scan和sink_scan传过来的数据。可以看一下是 receiveFromSingleReg 左表的时间长一些，还是右表的时间。

看着像probe阶段的 receiveFromSingleReg ，那个部分是在等待 table_scan 发过来的数据。这个中间有个runtime filter，不知道这块的流程是怎么样的。可以看看。

2、TableScan Node 的时间可能会包含： TableScan算子，Filter算子， Projection算子。可能要把这几个算子也加一下。

Jan 12 '24 11:01 ouyuanning

细分下算子的耗时。

根据 log, 在 Insert 期间，除了初期的几次 ranges 返回了 1000+ 个 block（非 fast ranges），其他时候都是 10 个以内。
table scan 每次的数据量大概不超过 0.5 mb
Insert 时，算子耗时总是一段时间高，一段时间又低一些。

|

@ouyuanning 麻烦看一下是不是正常的，预期的结果

Jan 15 '24 08:01 gouhongshen

和 @jensenojs @badboynt1 沟通后，发现可能是因为联合主键 filter 改写的逻辑问题。先 assign 给 @jensenojs 优化算子

Jan 19 '24 06:01 gouhongshen

比如： create table t1 (a int, b int, primary key(a,b)); insert into t1 values (1,1),(2,2),(3,3);

锦赛先比较一下这些方式的性能吧。（行数可以看下3 行， 30行， 300行，不同行数的情况） 1: (a=1 and b=1) or (a=2 and b=2) and or (a=3 and b=3) 2: pk in (serial(1,1), serial(2,2), serial(3,3)) 3: a in (1,2,3); 4: 留空，全表扫然后选性能好的方案就好了

Jan 19 '24 09:01 ouyuanning

其实你可以试一下直接用 select语句测，应该效果是一样的

Jan 19 '24 09:01 ouyuanning

还有一个。 filter_list 可能也可以根据情况来确定是否放进去。

Jan 19 '24 10:01 ouyuanning

@DanielZhangQD 现在插入的速度能够接受吗, 龙冉哥的相关pr已经merge进去了

Feb 18 '24 10:02 jensenojs

可能还需要继续做..?

Feb 22 '24 10:02 jensenojs

复现方式

use mo_catalog;

CREATE CLUSTER TABLE `statement_cu` (
`statement_id` VARCHAR(36) NOT NULL,
`account` VARCHAR(300) NOT NULL,
`response_at` INT DEFAULT NULL, # 修改了这个项的类型, 方便生成数据
`cu` int NOT NULL,
PRIMARY KEY (`statement_id`,`account_id`)
)

# 分批多次调用, 产生足够的数据
insert into test (statement_id, account_id, account, cu) select result || "cc", 1, '', 1 from generate_series (1, 3000000)g;

explain analyze INSERT INTO `mo_catalog`.`statement_cu` (`statement_id`,`account`,`response_at`,`cu`,`account_id`) VALUES ('0fa1346a-9aef-11ee-86a0-3acc4192c386','ba537a2b_43e7_4a2c_a6fe_b945278ef41a',1,1 ,24000002);

复现结果

企业微信截图_ff1bb3de-f94d-4148-890c-cab311ff8734

可以看到table scan读上来的数据已经被全部过滤掉了, 和之前的inputRows有对比

现在这个问题已经有很多pr在不同的层面上fix了, 而且相关的业务也已经被停掉了, 所以只能手动造数据来测, 已经跟海龙哥和远宁哥确认

我在dev环境上复制了一张statement_cu表出来, 然后模拟https://github.com/matrixorigin/MO-Cloud/issues/2200的方式往里面插入数据, 并用explain analyze 查看每个节点的耗时情况, 耗时最长的是table scan, 之前的慢有没有可能是s3的时延?

Mar 22 '24 08:03 jensenojs

matrixone
matrixone copied to clipboard

[Bug]: Inserting record is very slow

Is there an existing issue for the same bug?

Branch Name

Commit ID

Other Environment Information

Actual Behavior

Expected Behavior

Steps to Reproduce

Additional information

复现方式

复现结果

matrixone matrixone copied to clipboard

[Bug]: Inserting record is very slow

Is there an existing issue for the same bug?

Branch Name

Commit ID

Other Environment Information

Actual Behavior

Expected Behavior

Steps to Reproduce

Additional information

复现方式

复现结果

matrixone
matrixone copied to clipboard