paho.mqtt.java icon indicating copy to clipboard operation
paho.mqtt.java copied to clipboard

ClientState: C5180518000011: Timed out as no activity, keepAlive=60,000 lastOutboundActivity=1,534,923,567,979 lastInboundActivity=1,534,923,507,945 time=1,534,923,627,951 lastPing=1,534,923,567,979

Open chenyugui opened this issue 6 years ago • 52 comments

how to fix this issues! when i use the paho.mqtt.3.1.2.0 java.jar

chenyugui avatar Aug 23 '18 02:08 chenyugui

I think we'll need a bigger section of the trace. Looks like a keep alive timeout to me. What symptom does your application program get?

icraggs avatar Aug 23 '18 08:08 icraggs

my application with the issues is when the mqtt client publish a topic which has subscribed from the server, it will trigger this issues randomly.the issues title is the log. I use the version of the eclipse.paho java library as follow: 'org.eclipse.paho:org.eclipse.paho.client.mqttv3:1.2.0' the configuration of my application as follow: private boolean automaticReconnect = true; private int keepAliveInterval = 60; private int connectionTimeout = 30; private int maxInflight = Short.MAX_VALUE - 1;

chenyugui avatar Aug 23 '18 10:08 chenyugui

i meet this problem too,how do you fix this?

rexlin600 avatar Sep 17 '18 09:09 rexlin600

Maybe you can check the reasons from the MQTT server. @chenyugui I found that this timeout problem was due to the timeout caused by broker queue blocking between the server and the client when QOS = 2. But I'm not sure if I am right. @icraggs

rexlin600 avatar Sep 18 '18 01:09 rexlin600

Hi. I notice the same problem too: I have a java mqtt client. if my broker falls, the mqtt client gets a "connection lost", instead if I disconnect the router from the internet, the client just gets an error like

org.eclipse.paho.client.mqttv3.internal.ClientState checkForActivity SEVERE: CLIENT_ID: Timed out as no activity, keepAlive=20,000 lastOutboundActivity=1,537,783,810,402 lastInboundActivity=1,537,783,790,598 time=1,537,783,830,398 lastPing=1,537,783,810,402

but not connection Lost, so it behaves like it's still connected to mqtt, and did not try reconnection (nor authomatic nor from my java program). What should be the problem? I tried both 1.1.0 and 1.2.0 versions, same behavior.

lucapaciello avatar Sep 24 '18 12:09 lucapaciello

I am facing this issue too. will you please check below logs.

org.eclipse.paho.client.mqttv3.internal.ClientState.checkForActivity local server: Timed out as no activity, keepAlive=1,000,000 lastOutboundActivity=1,542,231,288,487 lastInboundActivity=1,542,230,318,179 time=1,542,232,288,486 lastPing=1,542,231,288,487 WARNING [MQTT Ping: local server] com.agile.microalley.mqtt.MqttListener.connectionLost connection lost WARNING [MQTT Ping: local server] com.agile.microalley.mqtt.MqttListener.connectionLost Timed out waiting for a response from the server

himeshagile avatar Nov 15 '18 06:11 himeshagile

I have the same problem with Mosquitto MQTT v3.1/v3.1.1 Broker as server . When my java application subscrible to a topic , after awhile the error message shows up :

Timed out as no activity, keepAlive=60,000 lastOutboundActivity=1,548,050,438,088 lastInboundActivity=1,548,050,408,304 time=1,548,050,498,088 lastPing=1,548,050,438,088

DennisGuo avatar Jan 21 '19 06:01 DennisGuo

So, this message means that a ping response hasn't been received from the MQTT server. The response on the part of the client library should be to close the connection at that point, and call the connectionLost callback if it is set.

The lack of response could be that the TCP socket has been broken but not reported to the application. This could happen if the TCP stack fails, or the MQTT broker fails to respond but keeps the TCP connection open. So this message is not necessarily the wrong behaviour.

If the connectionLost callback is not called as expected then this could be a problem. I'd need to see a full trace.

Ideally, a test would be added to the ConnectionLoss test suite for this situation (break the connection and wait for the keep alive processing to detect it). I'll add it at some point unless someone does it first (by PR).

icraggs avatar Jan 21 '19 14:01 icraggs

Test added. Seems to work.

icraggs avatar Jan 21 '19 17:01 icraggs

I've also seen this bug in 1.2.1. Once it happens, the client appears dead, i.e. threads keep on running but no more data is coming in. I've set connect option setAutomaticReconnect(true). A mosquitto_sub client running in parallel does keep receiving the data from the broker.

I can reproduce it as follows:

  • connect to a valid and reachable broker with connect option setAutomaticReconnect(true) and subscribe to a valid topic
  • publish something on the topic -> data arrives
  • cause a disruption on the network connection, e.g. disabling wifi on my laptop
  • wait until the keep alive period has passed -> the internal.ClientState.checkForActivity message appears
  • re-enable the network connection
  • publish something on the topic -> no data arrives at the paho client
  • wait some more (e.g. 10 minutes), publish again -> client still does not receive data

bertrik avatar Apr 24 '19 18:04 bertrik

Has some one fix this?

GitJacke avatar Apr 26 '19 04:04 GitJacke

I have the same problem,How to fix it up?

i402097836 avatar May 08 '19 05:05 i402097836

we have 6 publisher(mqtt client only for publishing the messages) behind the load balancer and we get below error only in 3 publisher. we are using java paho library 1.2.1

SEVERE: paho62104991016272: Timed out as no write activity, keepAlive=30,000,000,000 lastOutboundActivity=625,565,029,330,356 lastInboundActivity=625,555,607,648,593 time=625,645,608,992,578 lastPing=625,555,559,693,864

setting is following

#mqtt.connectiontimeout=60 #mqtt.keepaliveinterval=30 #mqtt.qos=0 #mqtt.cleanSession=true

conOpt.setAutomaticReconnect(true);

llakhch avatar May 10 '19 16:05 llakhch

EDIT: I wrote before that I might have encountered this before, but it could have been something else. Anyway, the rest of this post might be still of some use.

It's a bit of a long shot, but what if this reconnection failure is caused by the reconnection event running simultaneously to the disconnection event? Since ClientComms.shutdownConnection() can be run from multiple places (from the pinger, recv and send threads), what if it is run again just right after the reconnect thread is run?

I added a call to Thread.delay(500) at the start of shutdownConnection(), and another Thread.delay(1000) with a single one-shot recursive call at its end. This lines up the reconnect event (which runs 1000ms after the connection shutdown) with a potential second disconnect. Since Paho is disconnected at that point, it will never reschedule another reconnect operation and the client is left disconnected. I haven't seen it successfully reconnect in that test, but it could reconnect without the 500ms delay at the start of the function.

sp193 avatar May 13 '19 16:05 sp193

This might be a way to mitigate this condition, by adding a check for the disconnected state with isDisconnected(), to prevent a double-disconnection within shutdownConnection():

		// This method could concurrently be invoked from many places only allow it
		// to run once.
		synchronized(conLock) {
			if (stoppingComms || closePending || isClosed() || isDisconnected()) {
				return;
			}
			stoppingComms = true;

sp193 avatar May 13 '19 16:05 sp193

Any progress on this ?

sh0umik avatar Sep 13 '19 01:09 sh0umik

i meet this problem too,how do you fix this?

wangzengjing avatar Oct 11 '19 01:10 wangzengjing

Based on what's being said here https://www.eclipse.org/forums/index.php/t/1070780/, the client needs to reconnect after the keep-alive interval has passed. However, with the automaticReconnect flag set to true I expect the paho.MqttClient to do this for me.

For now I've circumvented this by catching the exception and calling .connect() again.

nickschriev avatar Nov 01 '19 14:11 nickschriev

@nickschriev If you can reliably reproduce the problem and have a working workaround, can you share your code?

bertrik avatar Nov 05 '19 10:11 bertrik

@bertrik

private void publish(byte[] payloadToPublish, MqttConnectOptions mqttOptions) {
     try {
          MqttMessage message = new MqttMessage(payloadToPublish);
          this.mqttClient.publish('myTopic', message);
        } catch (MqttException exception) {
          this.forceReconnect(mqttOptions);
        }
    }

And a forceReconnect() method like:

private void forceReconnect(MqttConnectOptions mqttOptions) {
    try {
        LOGGER.info("Enforcing a reconnect...");
        this.mqttClient.connect(mqttOptions);
        this.mqttClient.subscribe('myOtherTopic');
    } catch (MqttException mqttException) {
          LOGGER.error(String.format("Failed to force reconnect: %s", mqttException));
    }
}

Dont' forget to turn the automaticReconnect flag in the mqttOptions to false if using this workaround.

nickschriev avatar Nov 07 '19 16:11 nickschriev

ERROR o.e.p.c.mqttv3.internal.ClientState 210 - 172.18.215.24-gps-1: Timed out as no activity, keepAlive=200,000,000,000 lastOutboundActivity=15,938,114,406,087,056 lastInboundActivity=15,937,914,405,994,593 time=15,938,314,406,133,983 lastPing=15,938,114,406,088,182,

I meet this problem too. This problem occurs when use 1.2.0,1.2.1,1.2.2 versions. But This problem disappeared when set setCleanSession(true);

bobobo1114 avatar Dec 05 '19 03:12 bobobo1114

@bobobo1114 This message is logged when a ping message is not received from MQTT broker/server. With the latest develop branch, I do see this log message if server is not reachable from client.

Could you try to reproduce the problem with the latest develop branch and provide us with the log file with log level set to FINE?

rdasgupt avatar Apr 07 '20 07:04 rdasgupt

Same issue. After setting keepAlive to 0, the issue disappear too...

YiuTerran avatar Jun 09 '20 08:06 YiuTerran

Same issue. After setting keepAlive to 0, the issue disappear too...

In this solution, if broker disconnect, server can't auto-reconnect. So, now just use cleanSession=true or do reconnect by yourself

YiuTerran avatar Jun 11 '20 11:06 YiuTerran

This has been haunting me for as a long as I'm trying to use this library, with no clear solution in sight. The built-in auto-reconnect simply does not work, might as well just remove it to avoid giving people the wrong impression.

bertrik avatar Jun 13 '20 09:06 bertrik

I was wrong, sorry.

The problem I had was that my application was subscribed to a topic, then when a connection problem occurs (with a log as shown in the title) the client reconnects but messages are no longer coming in.

What I overlooked was that the subscription is not restored, even though the connection is. This wasn't obvious to me. I'm now relying on the built-in reconnect method of paho-mqtt (with exponential backoff, etc) and added an explicit callback that (re-)subscribes to the topic in the connectComplete method.

My now working code can be found here if you're interested: https://github.com/bertrik/LoraLuftdatenForwarder/blob/master/LoraLuftdatenForwarder/src/main/java/nl/bertriksikken/ttn/MqttListener.java#L78

bertrik avatar Jun 14 '20 12:06 bertrik

Very nice find @bertrik! Small but crucial nuance that probably took quite some time.

Big thanks for sharing your analysis!

nickschriev avatar Jun 15 '20 07:06 nickschriev

Ladies and gentlemen, this problem has also been encountered in my project. After a certain period of time, my java application client can't receive the Topic I subscribed to. [ ERROR o.e.p.c.mqttv3.internal.ClientState - xxx-server-z8g9: Timed out as no activity, keepAlive=20,000 lastOutboundActivity=1,596,179,662,984 lastInboundActivity=1,596,179,642,983 time=1,596,179,682,984 lastPing=1,596,179,662,984] How should I solve it? keepAlive=0? Or cleabSession=true? I'm confused. . .


By the way, the version I use is 1.1.1.

zhangdaopin avatar Jul 31 '20 07:07 zhangdaopin

@ITzhangdaopin the essence is that the connection to the MQTT server is restored by the automatic reconnect mechanism, but any subscriptions are lost. So you need to re-subscribe after a reconnect. In my example above, I'm doing this by subscribing to the topic in the connectComplete method of the callback handler.

bertrik avatar Jul 31 '20 08:07 bertrik

Bertrik Sikken is right here.Implements MqttCallbackExtended interface in ur mqtt client and override connectComplete.

@Override public void connectComplete(boolean reconnect, String serverURI) { logger.error("reconnect called :: " + reconnect); if (reconnect) subscribe(); <-- subscribe ur topics }

On Fri, Jul 31, 2020 at 1:33 PM Bertrik Sikken [email protected] wrote:

@ITzhangdaopin https://github.com/ITzhangdaopin the essence is that the connection to the MQTT server is restored by the automatic reconnect mechanism, but any subscriptions are lost. So you need to re-subscribe after a reconnect. In my example above, I'm doing this by subscribing to the topic in the connectComplete method of the callback handler.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/eclipse/paho.mqtt.java/issues/576#issuecomment-666992043, or unsubscribe https://github.com/notifications/unsubscribe-auth/AF3EIVMI2STAUXXTJBUFWXDR6J3ETANCNFSM4FRC46AA .

llakhch avatar Jul 31 '20 09:07 llakhch