Ratchet
Ratchet copied to clipboard
ratchetphp
Inconsistent latency sending data to client
I have 3 websocket servers running on the same hardware. When a message request comes in (onMessage), all 3 take less than 0.03s to process the the message from the client and create the correct response. However, as measured by Firefox/Chrome web inspector tools, one client gets its response in less than 200ms and the other two clients take 400+ms to get their responses. I am using the same computer to test, so server->client latency is consistent. The delay happens somewhere after the server interface class executes the ConnectionInterface $client->send command.
I am utilizing the uv extension for all three websocket servers (ExtUvLoop class)
I have no idea how to begin to troubleshoot this problem. Any advice on how to begin would be greatly appreciated.
@nivshah Why is this a problem for you? Also how many connections are those servers handling?
The servers are the interface for a bidding system that needs as low latency as possible for users. The connection numbers are quite low (less than 100) and ulimit has been expanded to allow a large number of connections.
I might not have the right model for my servers - they give updates whenever clients request them with an 'update' command rather than blasting updates out to the relevant clients. Just thinking through this, maybe whichever server has the most clients connected will naturally go slower since the server replies sequentially? Each client has some client-specific information in addition to highest bid, time remaining, etc. so that is why I have modeled the way I have.
My hypothesizing aside, I have one websocket server on this machine that consistently gives responses in 200ms or less even under high load, and one websocket server that is right now taking 400ms up to 2s to give a response under low load and I have no idea what the difference could be other than the port they are listening on behind a nginx reverse proxy. Implementation is basically identical between the two servers.
@nivshah Do you have any blocking code (maybe a PDO database call or something?)
FYI it doesn't matter if that call happens during a websocket operation, if it's there it blocks everything.
Both websocket servers make database operations, but both websocket servers finish their work in 1/100th of the total roundtrip time.
It feels a little bit like an OS-level problem, where one server is being invoked faster than the other. This is well outside my comfort zone so I'm again unsure what to even look for.
Both websocket servers make database operations, but both websocket servers finish their work in 1/100th of the total roundtrip time.
Are you using PDO or something like https://github.com/friends-of-reactphp/mysql for that?
It feels a little bit like an OS-level problem, where one server is being invoked faster than the other. This is well outside my comfort zone so I'm again unsure what to even look for.
There are a plethora of things that could be causing this. Among all things, it could be a broken network cable vein. One of the things you can start with is monitoring the response times and graph them out over time. This should give you insight if this is a time of day/week related or not.
I'm not using that MySQL extension, just using mysqli. I have a note to myself to implement this extension for some operations but I haven't gotten around to it.
Thanks for the ideas here, I'll see what makes sense.
mysqli blocks everything for the duration of that query, if you have a few quick ones that's not optimal but won't directly get you into trouble. But the more often queries you run that way the more often you block everything else, which can cause these delays
Do you recommend replacing all DB connections with this library or only the ones used by the websocket server?
All DB connections running in the process running the websocket server.