Sync data on MongoDB without copying existing docs on ES

[razlupercio]

The title is pretty self explanatory, we have used the meteor app, we have data from MongoDB on the ES server. Everything until here is okey. But, let's say, the meteor app stops and we have new data on the MongoDB server, how to sync this without duplicating data on ES?

I suppose I can use a transform function for this but I'm just lost on how to check the id from ES against the one on MongoDB. Any help is appreciated.

[Alino]

Hi raz, that is a good question. Currently I don't know. One option could be to flag every mongodb document with "wasMovedToES: true" and skip those from moving them to ES again?

or we could keep some log (new collection for this) in the db, of all already moved doc ids?

Maybe I will have to fight this issue as well in few weeks and some better idea will come to my mind.

[Alino]

I think maybe this could help https://www.elastic.co/guide/en/elasticsearch/guide/current/create-doc.html

if you change the way how mongo2es performs the POST or PUT request to te ES by adding op_type or _create to the url. It should not create duplicates if I understood.