disconnects, heartbeat_timeouts

[mateodelnorte]

Hi @alanxz,

Thanks for cooking up a great cpp rmq lib. We're spinning up multiple threads, which each create a channel/connection for sending to multiple queues. We were seeing connections go stale when inactive, so we forked and set BROKER_HEARTBEAT to the default 850. This fixed the stale connection issue, but resulted in rabbitmq correctly detecting and fully closing down the connections. Something is causing them to stop heartbeating after a certain amount of time, despite correctly starting off doing so.

Any ideas?

We end up with a number of sockets on the client in CLOSE_WAIT state, each of which has an accompanying heartbeat_timeout in sasl-log:

=ERROR REPORT==== 12-May-2015::17:36:36 ===
closing AMQP connection <0.14981.11> (127.0.0.1:51132 -> 127.0.0.1:5672):
{heartbeat_timeout,running}

=ERROR REPORT==== 12-May-2015::17:36:36 ===
closing AMQP connection <0.14978.11> (127.0.0.1:51131 -> 127.0.0.1:5672):
{heartbeat_timeout,running}

=ERROR REPORT==== 12-May-2015::17:36:36 ===
closing AMQP connection <0.14972.11> (127.0.0.1:51128 -> 127.0.0.1:5672):
{heartbeat_timeout,running}

=ERROR REPORT==== 12-May-2015::17:36:36 ===
closing AMQP connection <0.14999.11> (127.0.0.1:51134 -> 127.0.0.1:5672):
{heartbeat_timeout,running}

Appreciate the help.

Matt

[mateodelnorte]

Btw, this happens in deployed multi-machine scenarios as well as on localhost with all services and rabbitmq-server running locally.

Thx.

[alanxz]

Due to a limitation in the way rabbitmq-c works, the client must be in a blocking call waiting for the broker to send something in order to service heartbeats properly. IOW: you must be blocked in Channel::BasicConsumeMessage() if the program in question spends a lot of time waiting for messages.

Is this the case?

Also there have been some significant changes in the master branch of rabbitmq-c dealing with timeout code that hasn't had a chance to be vetted, so my current recommendation is to use rabbitmq-c v0.6.0 if possible.

[mateodelnorte]

We're actually not doing any consumes at all. This process simply takes messages coming in from another interface and publishes them to rmq.

I'll give a look to that version of rabbit-c unless you've got other ideas.

Thx On May 13, 2015 12:33 AM, "Alan Antonuk" [email protected] wrote:

Due to a limitation in the way rabbitmq-c works, the client must be in a blocking call waiting for the broker to send something in order to service heartbeats properly. IOW: you must be blocked in Channel::BasicConsumeMessage() if the program in question spends a lot of time waiting for messages.

Is this the case?

Also there have been some significant changes in the master branch of rabbitmq-c dealing with timeout code that hasn't had a chance to be vetted, so my current recommendation is to use rabbitmq-c v0.6.0 if possible.

— Reply to this email directly or view it on GitHub https://github.com/alanxz/SimpleAmqpClient/issues/101#issuecomment-101510982 .

[alanxz]

If you're not blocked on BasicConsumeMessage then heartbeats won't be serviced on that connection/channel, and thus if you don't send a message for a while the heartbeat will timeout and the connection will drop.

My concern was you were using some version of the master branch and there was some bug you might be hitting.

[mateodelnorte]

Does the master branch have heartbeating enabled outside that scenario? On May 13, 2015 1:30 AM, "Alan Antonuk" [email protected] wrote:

If you're not blocked on BasicConsumeMessage then heartbeats won't be serviced on that connection/channel, and thus if you don't send a message for a while the heartbeat will timeout and the connection will drop.

My concern was you were using some version of the master branch and there was some bug you might be hitting.

— Reply to this email directly or view it on GitHub https://github.com/alanxz/SimpleAmqpClient/issues/101#issuecomment-101518776 .

[alanxz]

The master branch adds the ability to service heartbeats when blocking on a send (i.e., when the broker has throttled clients because of memory pressure), this will be released as a part of version v0.7 of rabbitmq-c.

I'm working on adding full async support to the library at which point there will be full support for heartbeats in rabbitmq-c. (SimpleAmqpClient will likely need to be augmented to support the new APIs that will come of this).

[DanielBryars]

Hi Alanxz, for a send only scenario, is there a way for me to send the heart beat using your library from my code?

[alanxz]

Not from SimpleAmqpClient.

[shikharkhattar]

I'm facing a similar issue. I'm blocked on BasicConsumeMessage with default timeout (-1), but after a long time (> 12 hours maybe), if there are no messages in the queue, the ConsumerCancelledException is thrown. Is that the intended behaviour or is the broker closing the connection ?

What would be the best way to handle such large periods of inactivity ? I'm assuming the Async API (when it is released) would be more suitable for this.