redis-oplog
redis-oplog copied to clipboard
cult-of-coders
Feature/bulk redis payload processing
Ability to process redis payloads in bulk to reduce the number of requeries that large updates on a server can cause
Findings in production
Bulk updates that send many redis payloads to our meteor servers causes expensive database requeries for limit-sort publications and can spike cpu for other publications which attempt to process these payloads one at a time.
Goal
Reduce the large number of database lookups and requeries that happen when a large number of redis payloads are processed in bulk
Instead of doing document lookups and requeries one by one as the are needed, we process events in bulk, perform these database lookups in bulk and store them in a temporary documentMap and then iterate over the payloads triggering only a single requery when necessary.
This also gives us the ability to set maxRedisEventsToProcess threshold which prevents us from spiking server cpu for extended periods of time in the case where we have received too many redis payloads to process effectively.
Test Fixes
Looks like mocha tests had started failing due to some updates to the underlying mocha libraries in meteortesting:mocha-core. I pinned down the versions so that the issues with the test suite are resolved without needing to modify all the tests to no longer make use of async and done
Please respect the indentation as it's very hard for me to spot the differences. While I do agree that we should use 2, that's for another PR. For now we keep the standard.
@theodorDiaconu we've been using this in production for a few days now and are seeing great results. Great stuff @emaciel10
Findings in production
Bulk updates that send many redis payloads to our meteor servers causes expensive database requeries for limit-sort publications and can spike cpu for other publications which attempt to process these payloads one at a time.
What exactly do you mean by bulk updates? Like a Mongo initializeUnorderedBulkOp? I'm assuming it's not that as that can only be done on the raw collection beneath the scope of redis-oplog.
So if this for when doing a regular collection.update(query, update, {multi: true}) that would target a "bulk" amount of documents?
@evolross Yeah by "bulk updates" I am just referring to mongo updates that update a large number of documents. In some cases we use the raw mongo driver to trigger these updates and publish to redis after if really necessary but that is generally the exception. More often these are just regular collection multi updates when that update targets a very large number of documents
collection.update(query, update, {multi: true})
Hi @emaciel10 could you please pull the latest master into this PR, so that we can run the tests here on GitHub? Thank you!
@emaciel10 the tests are failing, please see here: https://github.com/Meteor-Community-Packages/redis-oplog/pull/9