do i need to explicitly close connections?

[orrgal1]

hi sometimes i notice that my process starts leaving many tcp connections in CLOSE_WAITING state. usually this happens when its processing a lot of messages off the queue. on a bad day this easily climbs up to the thousands. is there anything i should be doing to close the connections? is there something missing in the client to make sure the connections are closed? thanks

[edsrzf]

You should not have to close connections. If they're leaking, then it's a bug in the client.

Is there any more information you can provide that might help find the problem? A short program that reproduces would be ideal, but information about what calls you're making would help too.

[orrgal1]

Hi looked into Client.java and indeed disconnect() is never called on the connection. Probably because singleRequest() returns a Reader which cant be processed if the connection is closed. Maybe you guys should extend Reader and pass it back with a reference to the Connection so the connection can be closed after Reader is processed.

Reproducing should be easy. Just load some queue with about 20K messages. then run a program with 20 parallel threads that each take a message every 5 secs. make the processing also do something so that it puts some CPU and memory pressure on the machine. Keep pumping messages into the queue at a rate sufficient to keep it going for about 2 hours. After a while you should see the number of tcp connections in CLOSE_WAIT rising and not releasing.

On Sun, Sep 21, 2014 at 11:10 PM, Evan Shaw [email protected] wrote:

You should not have to close connections. If they're leaking, then it's a bug in the client.

Is there any more information you can provide that might help find the problem? A short program that reproduces would be ideal, but information about what calls you're making would help too.

— Reply to this email directly or view it on GitHub https://github.com/iron-io/iron_mq_java/issues/65#issuecomment-56310960.

[pax95]

Hi

There is some talk about this issue her http://stackoverflow.com/questions/4767553/safe-use-of-httpurlconnection http://stackoverflow.com/questions/3352424/httpurlconnection-openconnection-fails-second-time

A easy way of trying to solve the issue you could add conn.setRequestProperty("connection", "close"); in https://github.com/iron-io/iron_mq_java/blob/master/src/main/java/io/iron/ironmq/Client.java#L161

That should close and bypass the caching of the connection.

Best, Preben

2014-09-22 6:04 GMT+02:00 Or Gal [email protected]:

Hi looked into Client.java and indeed disconnect() is never called on the connection. Probably because singleRequest() returns a Reader which cant be processed if the connection is closed. Maybe you guys should extend Reader and pass it back with a reference to the Connection so the connection can be closed after Reader is processed.

Reproducing should be easy. Just load some queue with about 20K messages. then run a program with 20 parallel threads that each take a message every 5 secs. make the processing also do something so that it puts some CPU and memory pressure on the machine. Keep pumping messages into the queue at a rate sufficient to keep it going for about 2 hours. After a while you should see the number of tcp connections in CLOSE_WAIT rising and not releasing.

In any case I would appreciate it if you could let me know of your findings and if I should temporarily implement my own http calls until this is sorted out.

thanks!

On Sun, Sep 21, 2014 at 11:10 PM, Evan Shaw [email protected] wrote:

You should not have to close connections. If they're leaking, then it's a bug in the client.

Is there any more information you can provide that might help find the problem? A short program that reproduces would be ideal, but information about what calls you're making would help too.

— Reply to this email directly or view it on GitHub https://github.com/iron-io/iron_mq_java/issues/65#issuecomment-56310960.

— Reply to this email directly or view it on GitHub https://github.com/iron-io/iron_mq_java/issues/65#issuecomment-56326905.

[Stephenitis]

@pax95 should

conn.setRequestProperty("connection", "close");

be on by default on the client?

[pax95]

@Stephenitis It would be best if it could be configurable. Actually closing the connection would prevent the connection from being reused as a read it. Maybe the problem it is os or jdk dependent as the code seems ok. The reader is closed on every call even if the connection is not disconnected.

On the other hand it would be nice if connection params could be configurable eg. close connection, read- and connection timeout.

@orrgal1 you could try and see if conn.setRequestProperty("connection", "close"); fixes your problem ?

[pax95]

Just tried it out on a load test with 1 producer and 1 consumer, with and without conn.setRequestProperty("connection", "close"). I can not reproduce the issue with jdk 7 on osx, although setting the property seems to have a negative performance impact. I think you should create a load test with multi consumers/producers to see if the issue can be reproduced.

[orrgal1]

have not had a chance to try it out yet on a clean environment but if it helps my worker is running on an ec2 m1.large ubuntu machine. i have however downgraded my project to ironmq 0.0.8 in which i dont remember having this problem.

... and it looks like this version is behaving much better. i am currently processing with 5 threads and it looks like the number of open connections will rise to about 100 occasionally but then drop back to 9 or so. i'll wait a few more hours and update again.

... about 10 hours later and all is still well. so it looks like 0.0.8 is good. no leaking connections. does that help?

[orrgal1]

bad news.. the problem exists in 0.0.8 as well when processing volume gets high