canal icon indicating copy to clipboard operation
canal copied to clipboard

canal-adapter同步mysql至es,etlCondition无效

Open sherkwast opened this issue 4 years ago • 10 comments

Question

我的conf/es/test.yml内容如下:

dataSourceKey: defaultDS destination: example1 groupId: g1 esMapping: _index: test6-subaccount_history _type: _doc _id: _id upsert: true #pk: id sql: "select f_id as _id, f_id, f_version, f_user_id, f_account_id, f_subaccount_id, f_account_type, f_subaccount_type, f_action_type, f_currency, f_amount, f_balance, f_suspense, f_clearing, f_total, f_action_id, f_source_id, f_source_type, f_source_type_code, f_source_ext, f_created_at, f_updated_at from t_subaccount_history" #objFields:
#_labels: array:; etlCondition: "where f_version>3" commitBatch: 3000

不加etl条件可以正常的同步至es,但加上etl条件,依旧是把所有mysql更新或新增记录都同步到es,仿佛etl条件无效

sherkwast avatar Apr 30 '20 14:04 sherkwast

我也遇到了同样的问题

leeroy-code avatar Jun 18 '20 03:06 leeroy-code

Question

我的conf/es/test.yml内容如下:

dataSourceKey: defaultDS destination: example1 groupId: g1 esMapping: _index: test6-subaccount_history _type: _doc _id: _id upsert: true #pk: id sql: "select f_id as _id, f_id, f_version, f_user_id, f_account_id, f_subaccount_id, f_account_type, f_subaccount_type, f_action_type, f_currency, f_amount, f_balance, f_suspense, f_clearing, f_total, f_action_id, f_source_id, f_source_type, f_source_type_code, f_source_ext, f_created_at, f_updated_at from t_subaccount_history" #objFields: #_labels: array:; etlCondition: "where f_version>3" commitBatch: 3000

不加etl条件可以正常的同步至es,但加上etl条件,依旧是把所有mysql更新或新增记录都同步到es,仿佛etl条件无效

请问你这个问题解决了吗?

gangade avatar Sep 04 '20 01:09 gangade

etlCondition:条件无效

Setup-carlos avatar Nov 23 '20 12:11 Setup-carlos

同问

lufeng501 avatar Feb 19 '21 12:02 lufeng501

我也遇到了,1.1.5版本

SFight avatar Apr 15 '21 03:04 SFight

可能是yaml格式的问题,etlCondition字段需要在 esMapping子级

leos-code avatar Jul 23 '21 10:07 leos-code

也是醉了,这文档写的跟啥一样 etlCondition不支持直接写死的方式,如下 etlCondition: "where f_version>3"

只能这样etlCondition: "where f_version>{}" 然后再调用etl全量同步接口的时候 加上参数 ?params=3 这样才会生效

lyrric avatar Jul 30 '21 07:07 lyrric

也是醉了,这文档写的跟啥一样 etlCondition不支持直接写死的方式,如下 etlCondition: "where f_version>3"

只能这样etlCondition: "where f_version>{}" 然后再调用etl全量同步接口的时候 加上参数 ?params=3 这样才会生效

请问如果有多个条件呢?etlcondition 怎么写?还有etl指令怎么拼接参数?

axiaoqing avatar Aug 29 '22 07:08 axiaoqing

也是醉了,这文档写的跟啥一样 etlCondition不支持直接写死的方式,如下 etlCondition: "where f_version>3" 只能这样etlCondition: "where f_version>{}" 然后再调用etl全量同步接口的时候 加上参数 ?params=3 这样才会生效

请问如果有多个条件呢?etlcondition 怎么写?还有etl指令怎么拼接参数?

用分号分隔 etlCondition: "where id>={} and age = {}" curl http://127.0.0.1:8081/etl/rdb/mysql1/mytest_user.yml?params=4;44 -X POST

axiaoqing avatar Aug 29 '22 07:08 axiaoqing

那如果是增量呢?怎么处理过滤条件?

DemoLiang avatar Feb 04 '24 02:02 DemoLiang