elasticsearch-river-mongodb icon indicating copy to clipboard operation
elasticsearch-river-mongodb copied to clipboard

Need option to not skip oplog from migrating

Open nai0om opened this issue 10 years ago • 0 comments

Here is my use case : I have 10 shard of MongoDB with each shard have 100 GB. And I don't want to reindex all data every times river fail. So, I decided to create river directing to each shard (mean not connect to mongos). For this setup I will have 10 rivers running and if some river down I just reindex only on the shard that river working on and I not need to reindex all shard.

But for this use case river now not support this, then I fork project and change some code to make it works for my use case.

this is code that i remove:

// Initial support for sharded collection -
// https://jira.mongodb.org/browse/SERVER-4333
// Not interested in operation from migration or sharding
if (entry.containsField(MongoDBRiver.OPLOG_FROM_MIGRATE) && ((BasicBSONObject) entry).getBoolean(MongoDBRiver.OPLOG_FROM_MIGRATE)) {
    logger.debug("[Invalid Oplog Entry] - from migration or sharding operation. Can be ignored. {}", entry);
    return false;
}

This use case help me a lot.

Thanks

nai0om avatar Mar 27 '15 03:03 nai0om