nodeload icon indicating copy to clipboard operation
nodeload copied to clipboard

socket hang up problem with nodejs 0.8.2

Open Samuel29 opened this issue 12 years ago • 1 comments

Hi

while using nodeload accross many servers (dstributed onto EC2 instances), I saw strange bandwidth collapsing although my scenario was a simple "ramp up". After investigating, the collapsing was caused by some slaves who went into a strange state :

  • their cpu usage falls from 40% to almost 0%
  • they still send data to the master but values are zero...
  • in the logged output I see : ......WARN: Error during HTTP request: Error: socket hang up......... ................................
  • tcpdump -i eth0 -An port 80 shows me NOTHING, so the slave is no longer connecting

I think the socket reaches a timeout, but the reconnection code doesn't work, but I'm not a NodeJS expert .

here is my setup :

  • 1 master
  • 15 EC2 slaves running nodeload.js
  • the tested service is an amazon elastic load balancer with autoscaling group with apache/tomcat servers

the code that I use from the master :

var trk_test = {
        name: "30min ramp",
        host: targetHost,
        port: 80,
        timeLimit: 1800,
        loadProfile: [[0,0], [1800,500] ],
        numUsers: 1,
        stats: [/*..removed stuff..*/],
        requestGenerator: function(client) {
             /*..removed stuff..*/
            var request = client.request('GET', uri, headers);
            request.end();
            return request;
        }
};

console.log('\nStarting test.\n');
var cluster = new nl.LoadTestCluster('MYPUBLICIP:8601', slaveHosts);

// gracefully propagate the interruption to the slaves
process.on('SIGINT', function () {
        console.log('\nGot SIGINT. Stopping test\n');
        cluster.end();
});

cluster.run(trk_test);

cluster.on('end', function() {
        console.log('\nAll tests done. exiting.\n');
        process.exit(0);
});

Any idea of what happens ?

Samuel29 avatar Jul 12 '12 16:07 Samuel29

Hmm first look at the code :


util.createReconnectingClient = function() {
    var http = require('http'),
           /*(...)*/
            client = http.createClient.apply(http, clientArgs);
            client._events = util.extend(events, client._events); // EventEmitter._events stores event handlers
            client.emit('reconnect', oldclient);
        };

http://nodejs.org/docs/v0.8.2/api/all.html#all_http_createclient_port_host (http.createClient is deprecated)

so let's downgrade...

Samuel29 avatar Jul 13 '12 08:07 Samuel29