amazon-documentdb-tools icon indicating copy to clipboard operation
amazon-documentdb-tools copied to clipboard

Data inconsistency intermittently while using MVU-tool

Open sungsikkim82 opened this issue 6 months ago • 0 comments

I tested in my own lab for using MVU-tool. However, when I inserted a amount of data to the source and compare the collection count with the target, there are intermittent data inconsistencies.

From source Docdb, I was enabled the changed stream in specific database.

db.adminCommand({modifyChangeStreams: 1,
    database: "lab",
    collection: "", 
    enable: true});

And I running MVU-tool with current start-position.

python3 mvu-cdc-migrator.py \
--source-uri "mongodb://<<username>>:<<password>>@<<Source endpoint>>:27017/?tls=true&tlsCAFile=global-bundle-docdb.pem&replicaSet=rs0&retryWrites=false" \
--target-uri "mongodb://<<username>>:<<password>>@<<target Endpoint>>:27017/?tls=true&tlsCAFile=global-bundle-docdb.pem&replicaSet=rs0&retryWrites=false" \
--start-position 0166d02267000000030100000003000041e2 >> detailed_log2.txt

During migration, intermittent mismatch in the number of source and target data. However, after I killed mvu-cdc-migrator and restarted with same position(0166d02267000000030100000003000041e2), data was synced.

Would you please let me know what the root cause of this data discrepancy is? And would it be possible for mitigate this issue?

sungsikkim82 avatar Aug 29 '24 08:08 sungsikkim82