node-mongolian icon indicating copy to clipboard operation
node-mongolian copied to clipboard

How to run mapreduce on slave? i.e. how to set .setSlaveOk()

Open sajal opened this issue 13 years ago • 3 comments

Im trying Master/Slave replication for something. My goal is to use the slave only for mapreduce tasks, get the results inline and then store it into the master.

My full code is given below. When i use master for both serverm and servers , then it works, but if u use mapreduce on the slave server in servers, i get following error :-

> [debug] mongo://slaveserver:27017: Connected
[debug] mongo://slaveserver:27017: Initialized as unknown
[debug] Finished scanning... primary? no
{ stack: [Getter/Setter], arguments: undefined, type: undefined, message: 'Could not connect to primary: ' }
[debug] mongo://slaveserver:27017: Disconnected

AFAIK mongo will allow you to run mapreduce on slave as long as the output is inline and setSlaveOk() is set. How do i set that using mongolian?

in the code below, dbs does not have a method getMongo()

var mongofunctions = require('./mongofunctions'),
  conf = require('./config/conf'),
  helpers = require('./helpers'),
  Mongolian = require("mongolian"),
  serverm = new Mongolian("masterserver:27017"),
  servers = new Mongolian("slaveserver:27017"),
  dbm = serverm.db(conf.dbname),
  dbs = servers.db(conf.dbname),
  beacons = dbs.collection("beacons"),
  hourlystats = dbm.collection("hourlystats"),
  params = {};

var start = new Date((new Date()).getTime() - 1 * 60 * 60 * 1000 ); // 1 hour ago
start.setUTCSeconds(0);
start.setUTCMilliseconds(0);
start.setUTCMinutes(0);
var end = new Date(start.getTime() +  1 * 60 * 60 * 1000); // start plus an hour

params.timestamp = {
  "$gte": start,
  "$lt": end
};

beacons.mapReduce(
  mongofunctions.cronstatgenerator.map,
  mongofunctions.cronstatgenerator.reduce,
  {
    out: {inline:1},
    finalize: mongofunctions.cronstatgenerator.finalize, 
    query: params
  }, function (error, result) {
    console.log(error);
    result.results.forEach(function(item){
      //console.log(item);
      hourlystats.save(item);
    });
    process.exit();
  }
)

sajal avatar Jan 08 '12 14:01 sajal

Would like to add... that i can run this mapreduce with inline output on the slave using the mongo console directly without any issues.

sajal avatar Jan 08 '12 14:01 sajal

Mongolian doesn't support slave servers yet, so right now your only option would be to hack the Mongolian source.

marcello3d avatar Jan 08 '12 23:01 marcello3d

Aha thanks... it turns out that my map-reduce job was taking hours to finish cause amazon throttles down micro instances big time, even when run from the mongo shell. So for now im doing my processing client side using python.

sajal avatar Jan 10 '12 09:01 sajal