Changing Client IP/origin to secondary IPs

[MasterJames]

I finally realized what's been causing me problems in a new improved iteration of my personal project. Needing multiple CPUs, I've forked nodejs instances and having the second instance use another secondary IP attached to the same interface. When it goes to connect as a client (server to server communication), I can try to pass the real alternative secondary IP the client 'request' but as it then goes to the 'connect' method it still has the default primary ip of the machine, and not the secondary associated with the process. I've tried setting 'origin' (connects 3rd argument) and I see it in the request, alas it's not changed the remoteAddress in the connection and messages the follow. How can I tell the client which IP it is using? I've noticed the remoteAddresses where there is an array for multiple addresses so it feels like this was never properly addressed and I'm not sure where I can resolve the problem. https://github.com/theturtle32/WebSocket-Node/blob/master/lib/WebSocketClient.js I'm not interested in using different ports, it's not an option. I fear I'm going to have to get the primary server to establish all local secondary IP connections as it simple seems there is no way to make a connection from a client process that is running a secondary IP address, and set which IP shows up in the remoteAddress of the servers connection. I maybe able to pass the request origin to the socket before sending the request call which goes to the connect and then there is a value passed through there. How can this be better? Is it not suppose to change the remoteAddress or append the origin to remoteAddresses? I know it's complicated, just ask for clarity if you need it, please. It appears to me now that lots of questions online are actually this problem but nobody has figured that out yet. It seems a critical issue to me.

[MasterJames]

So maybe this idea is working for me?

	request: function ( req ) {
		req.socket.ip = req.origin;

later he wrote...

	connect: function ( conn ) {
		console.log(conn.socket.ip);

and of course the 3rd argument passes as origin...

	conn.connect( 'wss://' + ip + '/urlQuery', 'protocol', target.ip );

Am I missing something else obvious here? Hope it helps, it was hard to track down due to extra added complexities I suppose.

Still I pose the question; What's the best way to set the IP of a client to a custom, alternative, secondary IP ?

[MasterJames]

No I've looked at this over and over a few times and it's an issue let me clarify. It seems if you launch a process for hosting on another secondary ip (on the same machine), the client connection made to the same machine (but primary ip [for server to server link) or any other local machine running a server, and then that server sees it's remoteAddress (and all other places) as the primary base ip for the machine (not it's secondary ip, it represents). We need to be able to tell from the server (at request and connect), when the client is connecting which secondary IP the client is calling from. Therefore I guess be able to set the remoteAddress (to a valid secondary IPv4/IPv6 address). Again it seems the client connection object takes and sends as the primary ip for all forked or other cpu processes (also server) on that machine. Is there a way to set this parameter on a unix level per process or something I'm missing as a possible work around?, and where it the code would we find the way to re-address the client connection (suggested PR target)? Finally... Is this a Bug, or feature request or an I missing something more obvious? Please advise, Thanks.

[MasterJames]

I've noted since that by passing the 3rd argument with an object of headers, that X-Real-IP is a better approach, then messing with the origin (prior argument). {"x-real-ip": ws.ip } and then you'll see it like so. console.log("request headers:", req.httpRequest.headers["x-real-ip"]);

[rgillan]

Just trying to do the same with a client test set up environment where we need to generate 100's of k of client connections. Working with primary IP (64k clients) no problem. Trying to set localAddress in the client.connect extraRequestOptions is silently failing (no clients started up).

var Stage1Client = new WebSocketClient(); Stage1Client.connect(c.stage1Protocol + '://' + c.stage1UriP + ':' + c.stage1Port + '/', c.stage1SubProtocol, c.stage1Origin, {}, {localAddress: interfaces[1]});

where interfaces is an array of local IP addresses on the AWS EC2 instance [primaryIP, secondaryIP1, secondaryIP2...]

Have logged at line ~232 on WebSocketClient.js, after the passed in options have been extended into requestOptions, and it looks like it extends the object correctly:

requestOptions:{"agent":false,"localAddress":"172.31.9.49","hostname":"nisient.net","port":"443","method":"GET","path":"/","headers":{"Upgrade":"websocket","Connection":"Upgrade","Sec-WebSocket-Version":"13","Sec-WebSocket-Key":"XBvh2iPvzueGd4JNhDkc5g==","Host":"nisient.net","Sec-WebSocket-Protocol":"s1.nisient.net","Origin":"nisient"}}

This is the first of the active secondary addresses on the instance. Our only other option is to spin up multiple test AWS instances each doing 64k connections although this is obviously less preferable. We're on node 10.9.0

[rgillan]

Been able to isolate this in AWS world. tcpdump showed that indeed node.js was correctly selecting the localAddress for the output interface, but wasn't hitting the destination from the virtual IP on the client swarm device. All sorted it seems

[rgillan]

still hitting a port limit when the combined port count hits ~64k (ie. 32k on each interface). Not sure if this is an AWS or somewhere buried in the ubuntu kernel (we already tune fd limits, local port ranges, tcp buffer stuff). Consistent with ~21k on 3 interfaces so it's a limit somewhere if anyone has a clue (not really the right place to be asking this)

[MasterJames]

Not sure if it makes any difference but I recall adding a way that it splits out for each CPU a nodejs instance that gets its own IP and IPV6 address. So with 4 CPUs I get 3 instances running so 6 addresses (3 IPs + 3 IPV6s) per EC2 instance. Maybe you have to build the kernel but there is stuff about getting 600K connections here.

https://blog.jayway.com/2015/04/13/600k-concurrent-websocket-connections-on-aws-using-node-js/

[rgillan]

Thanks for getting back. The client side is working ok with secondary IPs. Issue was on the total connection count on the server side, and even though we'd adjusted the limit upwards it appears that even though we're using secondary IPs the AWS server side sees them coming from the same address (mac address at least). We've run up multiple instances of our swarm tester so all good. Cheers