MongoShake icon indicating copy to clipboard operation
MongoShake copied to clipboard

配置白名单 applyOps类型无法过滤

Open liuxinwang opened this issue 2 years ago • 2 comments

问题:配置了白名单,还是有其它库表数据发送到kafka,debug观察日志: [2023/02/12 20:00:45 CST] [DEBG] NamespaceFilter check oplog:[{applyOps [[{op u} {ns platform_core_order.order_after_sale} {ui......

版本:v2.8.2

配置文件:

conf.version = 10
id = mongoshake
master_quorum = true
full_sync.http_port = 9101
incr_sync.http_port = 9100
system_profile_port = 9200
log.level = debug
log.dir =
log.file = collector.log
log.flush = false
sync_mode = incr
mongo_urls = mongodb://root:[email protected]:3717,xxxxxx.mongodb.rds.aliyuncs.com:3717/

mongo_cs_url =
mongo_s_url = 
mongo_ssl_root_ca_file = 

tunnel = kafka
tunnel.address = mongo_oplog_local_dev@xxxxxx:9092
tunnel.message = json
tunnel.kafka.partition_number = 12
tunnel.json.format =
tunnel.mongo_ssl_root_ca_file =

mongo_connect_mode = secondaryPreferred

filter.namespace.black =
filter.namespace.white = dba_test
filter.pass.special.db =
filter.ddl_enable = false
filter.oplog.gids = false

checkpoint.storage.url =
checkpoint.storage.db = mongoshake_dev
checkpoint.storage.collection = ckpt_default
checkpoint.storage.url.mongo_ssl_root_ca_file = 
checkpoint.start_position = 2023-02-08T00:03:51Z

transform.namespace =

full_sync.reader.collection_parallel = 6
full_sync.reader.write_document_parallel = 8
full_sync.reader.document_batch_size = 128
full_sync.reader.fetch_batch_size = 8192
full_sync.reader.parallel_thread = 1
full_sync.reader.parallel_index = _id
full_sync.collection_exist_drop = true
full_sync.create_index = none
full_sync.executor.insert_on_dup_update = false
full_sync.executor.filter.orphan_document = false
full_sync.executor.majority_enable = false

incr_sync.mongo_fetch_method = oplog
incr_sync.change_stream.watch_full_document = false
incr_sync.oplog.gids =
incr_sync.shard_key = auto
incr_sync.shard_by_object_id_whitelist = 
incr_sync.worker = 12
incr_sync.tunnel.write_thread = 12
incr_sync.target_delay = 0
incr_sync.worker.batch_queue_size = 64
incr_sync.adaptive.batching_max_size = 1024
incr_sync.fetcher.buffer_capacity = 256
incr_sync.reader.fetch_batch_size = 8192
incr_sync.executor.upsert = false
incr_sync.executor.insert_on_dup_update = false
incr_sync.conflict_write_to = none
incr_sync.executor.majority_enable = false
special.source.db.flag =
skip.nsshardkey.verify =

liuxinwang avatar Feb 13 '23 02:02 liuxinwang

applyOps是嵌套的,之后版本修正

zhangst avatar Feb 13 '23 03:02 zhangst

请问这个修复了吗,配置白名单,仍然会同步其他db

datuxing avatar Nov 16 '23 02:11 datuxing